Its author, Prof Mathieu Ossendrijver, from the Humboldt University of Berlin, Germany, said: "I wasn't expecting this. It is completely fundamental to physics, and all branches of science use this method."
The Ancient Babylonians once lived in what is now Iraq and Syria. The civilisation emerged in about 1,800 BC.
Clay tablets engraved with their Cuneiform writing system have already shown these people were advanced in astronomy.
"They wrote reports about what they saw in the sky," Prof Ossendrijver told the BBC World Service's Science in Action programme.
"And they did this over a very long period of time, over centuries."
But this latest research shows they were also way ahead when it came to maths.
It had been thought that complex geometry was first used by scholars in Oxford and Paris in medieval times.
They used curves to trace the position and velocity of moving objects.
But now scientists believe the Babylonians developed this technique around 350 BC.
Prof Ossendrijver examined five Babylonian tablets that were excavated in the 19th Century, and which are now held in the British Museum's archives.
The script reveals that they were using four-sided shapes, called trapezoids, to calculate when Jupiter would appear in the night sky, and also the speed and distance that it travelled.
"This figure - a rectangle with a slanted top - describes how the velocity of a planet, which is Jupiter, changes with time," he said.
"We have a figure where one axis, the horizontal side, represents time, and the other axis, the vertical side, represents velocity.
"The area of trapezoid gives you the distance travelled by Jupiter along its orbit.
"What is so special is this type of graph is unknown from antiquity - so making figures of motion in this rather abstract space of velocity against time - this is something very, very new."
He added that there was evidence that the Greeks used a "more straightforward" form of geometry, which dealt with the spatial relationships between the Earth and the planets rather than the concepts of time and velocity.
Prof Ossendrijver told the BBC that it was unclear how common this technique was.
"It could be that there was an earlier tablet, written by a genius, by one individual, who came up with this new way of doing astronomy.
"It could also be that in fact this is a method that was more widely applied by different scholars. We don't know."