CSCI 341 Theory of Computation

Fall 2025, with Schmid

Computation in a Nutshell

Computer Science is one of those rare fields of study \(X\) where the question "what is \(X\)?" actually has a pretty satisfying answer. We asked this exact question, where \(X = \) computation, in the first section of the course, but we weren't really ready for that satisfying answer. Now we are (kind of)!

Question: what is computation?

Computation is the manipulation of semantical objects by syntactic means.

...ehhhh ok, maybe we weren't quite ready for this yet. Today we are going to take the first couple of steps to understanding the bolded statement above, and talk about what "semantical objects" are and what "syntax" is.

Syntax and Semantics

"Syntax" is a word that comes from linguistics. In that context (and in the context of programming languages, too), the syntax of a language consists of the strings of text that are "well-formed". In linguistics, "well-formed" might mean "grammatical", and similarly in a programming language it might mean that there are no syntax errors (you haven't forgotten a bracket or a colon or something). Abstractly, for us, syntax really just refers to strings of text, or words as we have been calling them.

Part of the point of syntax is that it is completely devoid of meaning. It's just text! The very nature of computation demands that when we get a machine to compute something, it should never need to understand what it is doing. Because it's not going to. It's a machine, after all. It should be able to get its job done by just pushing symbols around.

Automata are a good examples of "machines that just push text around". Expanding our definition of "syntax" slightly, running an automaton just involves placing and removing "active" markers (see Reading Words) according to some set rules. It is in this sense that automata calculate.

But computation wouldn't amount to much if it was just text being pushed around. To us, the observer, each bit of syntax means something. The meaning we ascribe to a string of text, i.e., our interpretation of it, is its semantics. That type of thing that you interpret the text to mean is what is being referred to as a "semantical object".

Semantical objects can be as simple and finite, or as complex and infinite, as you want. For a concrete example, consider the regular expression \((a + ((b \cdot c))^*)\). Before you knew what all the symbols meant, it was just a string of text, nothing more, and isn't even that long a string of text (21 characters). But how we interpret that string of text, i.e., its semantics, is as a language: it represents the language \[ \mathcal L((a + ((b \cdot c))^*)) = \{a(bc)^n \mid n \in \mathbb N\} \] That's not finite at all! Somehow, we were able to capture this entire language with just 21 characters.

This is where computation gets its power from: text is easy to work with in comparison to infinite sets of numbers, words, etc.! But the simplicity of text also places a limit on what computation can and cannot accomplish. In this chapter of the course, on computability, we are going to prove, mathematically, that there is an upper bound on what computation can achieve. In fact, we will see a whole host of problems that cannot be solved computationally.

Mathematical proof requires rigour, so let's get started on that. We are going to start by taking a look at representation, which is the process of taking something and turning it into syntax. The opposite direction, turning a piece of syntax back into its semantics, is called interpretation.

Representations of Decision Problems

The problems we have learned to solve with the models of computation we have seen so far in the course have only dealt with deciding language membership, i.e., deciding if a word is an element of a given language. This is not so dramatic a restriction: in fact, just about any decision problem (deciding between "yes" or "no") can be translated into a language membership decision problem (deciding whether a word is in a language or not). This fact is essential to modern computing. In your computer, every piece of information that is stored is represented as a bitstring, i.e., a word \(w \in \{0,1\}^*\). Therefore, every decision problem that can be solved with a computer program must boil down to a decision problem in bitstrings! Let us make all of this more general and more precise.

(Representation of Decisions) Let \(S\) be any set. Formally, a decision problem in \(S\) is just a subset \(D \subseteq S\), often described as a property of elements of \(S\) (we can translate between the two by taking \(D\) to be the set of elements of \(S\) that satisfy the property).

Now let \(A\) be an alphabet. A string representation of the elements of \(S\) in the alphabet \(A\) is a function that takes an element of \(S\) as input and returns a string, \( \rho \colon S \to A^* \).

Given a decision problem in \(S\), \(D \subseteq S\), and a pair \((f, L)\), where \(\rho \colon S \to A^*\) is a string representation of the elements of \(S\) and \(L \subseteq A^*\) is a language, we say that \((f, S)\) is a faithful representation of \(D\) if

for any \(s \in S\), \(\rho(s) \in L\) if and only if \(s \in S\), and
\(\rho\) is injective.

In a diagram,

(Even-numbers) Consider the following familiar decision problem in \(\mathbb N\).

The Even-numbers Problem. Given a natural number \(n \in \mathbb N\), is \(n\) a multiple of \(2\)?

Formally, this would be represented as \(E = \{2n \mid n \in \mathbb N\} \subseteq \mathbb N\).

There are different ways to faithfully represent the even-number problem. What we have done in the past, in this course, is simply to represent every number \(n\) as a string of \(a\)'s (where \(a\) is some letter) of length \(n\). That is, we represent \(n\) with \(a^n\). This defines the string representation of numbers below: \[ \rho \colon \mathbb N \to \{a\}^* \qquad \rho(n) = a^n \] Representing numbers this way paints a very clear picture of how to find a language that faithfully represents the even-numbers problem. Here it is in words:

The \(\mathcal L((aa)^*)\) Membership Problem. Given a string of \(a\)'s \(a^n \in \{a\}^*\), is \(a^n \in \mathcal L((aa)^*)\)?

According to our formal definition, the pair \((\rho, \mathcal L((aa)^*))\) is a faitful representation of the even-numbers problem.

Another classic string representations of natural numbers is one we use one all the time! \[\begin{gathered} \mathsf{dec} \colon \mathbb N \to \{0,1,2,3,4,5,6,7,8,9\}^* \\ \mathsf{dec}(n) = \text{(the shortest decimal expansion of \(n\))} \end{gathered}\] Note that the word "shortest" means that \(\mathsf{dec}(n)\) only starts with a \(0\) if \(n = 0\). There are analogous representations for binary, hexadecimal, and so on.

(Even Binaries and Trinaries) Consider the binary representation of natural numbers \[\begin{gathered} \mathsf{bin} \colon \mathbb N \to \{0,1\}^* \\ \mathsf{bin}(n) = \text{(the shortest binary expansion of \(n\))} \end{gathered}\] For each of the decision problems \(D \subseteq \mathbb N\) below, find a regular expression \(r\) such that \((\mathsf{bin}, \mathcal L(r))\) is a faithful representation of \(D\).

\(D_1 = \{n \in \mathbb N \mid n > 0\}\)
\(D_2 = E = \{n \in \mathbb N \mid \text{\(n\) is even}\}\)
\(D_3 = \{2^n \mid n \in \mathbb N\}\)
\(D_4 = \{2^n + 1 \mid n \in \mathbb N\}\)

Along the way, you might need to "solve" the problem, in the sense that you might have to draw an automaton with a state that accepts the language.

(5 out of 16) Consider the decision problem \[ D_5 = \{5n \mid n \in \mathbb N\} \subseteq \mathbb{N} \] Given a natural number \(n\in \mathbb N\), let \(\mathsf{hex}(n)\) be the hexidecimal representation of \(n\). Find a regular expression \(r\) such that \((\mathsf{hex}, \mathcal L(r))\) is a faithful representation of \(D_5\).

Do this first for the numbers \(2\) and \(3\) (even in base \(3\)) instead of \(5\) and \(16\) first, then try it for \(3\) and \(10\) (multiple of \(3\) in decimal). This is going to help with the second one: Divisibility Rule for 3 on GeeksForGeeks.

The definition of faithful representation is very general: it allows us to extend our definitions of "regular" and "context-free" to other types of decision problems.

(Families of Problems) Let \(\mathsf{Fam}\) be a family of languages in the alphabet \(A\), and let \(S\) be any set. A decision problem \(D \subseteq S\) is said to be in \(\mathsf{Fam}\) if there is a faithful representation \((\rho, L)\) of \(D\) with \(\rho \colon S \to A^*\) and \(L \subseteq A^*\).

For example, we saw that \(D_1,D_2,D_3,D_4\), and \(D_5\) above are all regular problems (in the sense that they are in \(\mathsf{Reg}\)).

In the previous exercises (and problem), you found faithful representations of a handful of number-theoretic problems. Other types of objects have string representations though. Sometimes these take a bit more work and ingenuity.

(Representing Order) Let \(S\) be the set of all complete binary trees where the data of the nodes of the tree are numbers from the set \(A = \{0,1,2,3,4,5\}\). Let \(D \subseteq S\) be the set of all trees \(t \in S\) such that \(t\) is a binary search tree. Find a faithful string representation of \(D\). Is \(D\) a regular problem?

Try an in-order traversal of the tree!

We are not quite ready to get into what exactly a solution to a decision problem is yet. For now, we are going to let ourselves be content with the idea that a solution to a decision problem \(D\) is a faithful representation \((f, L)\) of \(D\) and a finite automaton (of some kind, maybe with a stack or a counter!) \(\mathcal A\) with a state \(x\) such that \(L = \mathcal L(\mathcal A, x)\). We will be more precise about this later.

(3 out of 2) Let \(D = \{3n \mid n \in \mathbb N\}\). Find a language \(L \subseteq \{0,1\}^*\) such that \((\mathsf{bin}, L)\) is a faithful representation of \(D\). Find a stack automaton \(\mathcal S\) with a state \(x\) such that \(L = \mathcal L(\mathcal S, x)\).

Representations of Problems Generally

So we've talked about decision problems a lot. But not every computational problem is a decision problem. For example, given \(n \in \mathbb N\), how does one compute \(n^2\)? This is not so much a decision problem as it is a finding-a-function problem. These more general types of problems are also going to be useful for us to understand from a computational perspective, so we should spend some time with them here.

(Representation of Functions) Let \(S_1\) and \(S_2\) and be any sets, and let \(f \colon S_1 \to S_2\) be any function. Given string representations \(\rho_1 \colon S_1 \to A^*\) and \(\rho_2 \colon S_2 \to A^*\), we say that \((\rho_1, g, \rho_2)\) is a representation of \(f\) if \(g \colon A^* \to A^*\) and for any \(s \in S_1\), \[ g(\rho_1(s)) = \rho_2(f(s)) \] and \(\rho_1,\rho_2\) are injective. That is, computing the functions one at a time from top-left to bottom-right in the diagram below always ends at the same word both ways.

In a diagram,

(Taking Negatives) Consider the following string representation of integers \(\rho \colon \mathbb Z \to \{0,1\}^*\), defined by \[ \rho(n) = \begin{cases} 0~\mathsf{bin}(n) &\text{ if \(n \ge 0\)} \\ 1~\mathsf{bin}(n) &\text{ if \(n \le 0\)} \end{cases} \] For example, \[ \rho(0) = 00 \quad \rho(1) = 01 \quad \rho(-1) = 11 \quad \rho(-2) = 110 \] We would like to represent the function \(f \colon \mathbb Z \to \mathbb Z\) that takes the negative of every number, \[ f(n) = -n \] Here is one such representation: define \(g \colon \{0,1\}^* \to \{0,1\}^*\) by \[ g(w) = \begin{cases} 0~u &\text{ if \(w = 1u\)} \\ 1~u &\text{ if \(w = 0u\)} \\ \varepsilon &\text{ if \(w = \varepsilon\)} \end{cases} \] This function flips the first bit of the input word if it is nonempty and sends the empty word to itself. To see that it is a representation of \(f\), we need to check that \(g(\rho(n)) = \rho(f(n))\) for all \(n \in \mathbb Z\). If \(n \ge 0\), then \[\begin{aligned} g(\rho(n)) &= g(0~\mathsf{bin}(n)) \\ &= 1~\mathsf{bin}(n) \\ &= \rho(-n) \\ &= \rho(f(n)) \end{aligned}\] The case where \(n < 0\) is very similar.

(Finishing the Negative Example) In the Taking Negatives example, we defined a representation of integers \(\mathbb Z\) and functions \(f\) and \(g\). Show that if \(n < 0\) is any negative integer, then \(g(\rho(n)) = \rho(f(n))\).

(Multiplying by Two) Find a representation \((\mathsf{bin}, g, \mathsf{bin})\) of the function \(f \colon \mathbb N \to \mathbb N\) defined by \(f(n) = 2n\), where \(\mathsf{bin} \colon \mathbb N \to \{0,1\}^*\) is the binary representation function.

(Composing Representations) In this problem, we are going to show that representations of functions "compose". We need a bit of notation: given functions \(f_1 \colon S_1 \to S_2\) and \(f_2 \colon S_2 \to S_3\), we are going to define \(f_2 \circ f_1 \colon S_1 \to S_3\) to be the function defined by \[ f_2 \circ f_1(s) = f_2(f_1(s)) \] This function, \(f_2 \circ f_1\), is called the composition of \(f_1\) and \(f_2\).

Now on to the problem. Let \(S_1,S_2,S_3\) be sets and let \(A\) be an alphabet. Let \(\rho_i \colon S_i \to A^*\) be a string representation for each \(i = 1,2,3\), and let \(f_1 \colon S_1 \to S_2\) and \(f_2 \colon S_2 \to S_3\) be functions. \[ S_1 \xrightarrow{f_1} S_2 \xrightarrow{f_2} S_3 \] Given a representation \((\rho_1, g_1, \rho_2)\) of \(f_1\) and a representation \((\rho_2, g_2, \rho_3)\) of \(f_2\), prove that \((\rho_1, g_2 \circ g_1, \rho_3)\) is a representation of \(f_2 \circ f_1\).

This lecture was all about what it means to represent everyday computational problems in the format of strings, which is what all of our models of computation so far have operated on. Furthermore, the models of computation we have been working with up until now were really only able to decide. What we are missing are models of computation that can do what functions do: take input and produce output. In the next lecture, we will look at a simple format of computable functions, which are functions of the form \(g \colon A^* \to A^*\) (like in the definition of function representations) that are defined "via syntactic means".

← cfl pumping lemma lambda calculus →

Top