The following events are significant both to the history of the Department and to the narrower scientific areas with which it is concerned. In 1956, the Computing Center was founded at the Department of Mathematics of the then Faculty of Science and Mathematics. Already two years later, in 1958, the course Numerical Analysis was introduced as a compulsory course for all mathematics students. The Department of Numerical Mathematics and Programming, headed by Prof. Dr. Konstantin Orlov, was founded in 1971. In the following year, the study program B - Numerical Mathematics with Cybernetics was also established. In 1981 this study program changed its name to Numerical Mathematics, Cybernetics and Optimization, since 1990 to Numerical Mathematics and Optimization, and since 2009 until today it exists as the Applied Mathematics module in the Mathematics program. The Department officially received its current name in 1987 after the set off of the Department of Computer Science.

The heads of the Department of Numerical Mathematics and Optimization were: prof. Dr. Boško Jovanović, prof. Dr. Arif Zolić, prof. Dr. Vladimir Janković and prof. Dr. Milan Dražić.

The department is responsible for teaching activities in the field of applied mathematics at the Faculty of Mathematics, with special emphasis on the following areas: Numerical Mathematics, Optimization, Graph Theory, and Mathematical Modeling. Members of the department also teach mathematical subjects at other faculties of the College of Belgrade.

In the period from 1971 to the present, more than 40 doctoral theses and a large number of master's theses have been defended at the Department.

In the 1980s, the Department maintained a Seminar of Numerical Mathematics. Today, members of the Department actively participate in the work of the Seminar of Computer Science and Applied Mathematics of the Institute of Mathematics of the Serbian Academy of Sciences (headed by Prof. Dr. Vera Kovačević-Vujčić and Prof. Dr. Milan Dražić). Members of the Department have published a large number of monographs and scientific papers, some of which have appeared in the most important international journals. The Department was several times the organizer of the international symposium on Operational Research SYM-OP-IS.