Ludwig Wittgenstein is often described as one of the greatest philosophers of the 20th century and the author of the first comprehensive theory of linguistic reference. Since Wittgenstein is often perceived this way, his theses are usually compared with other philosophers' ideas concerning language as if they were absolutely new. However, some ideas very similar to Wittgenstein's were first formulated much earlier. One of the most prominent pedagogical Augustine's concepts implied that it is impossible to teach someone words using words as well as signs using signs. Nevertheless, Wittgenstein argued with Augustine in one of his works as if the letter was the supporter of ostensive theory. After revealing some historical misunderstandings it becomes clear that there is no contradiction between two great philosophers on these general questions. However, a closer study shows that there is more to the question than it usually seems and there are several important points preceding the concepts of Augustine and Wittgenstein.