The US Secret Service was engulfed in a sex scandal on Saturday after a dozen agents assigned to President Barack Obama's trip to Colombia were sent home amid reports of them using prostitutes.

Five US military personnel are also being investigated for misconduct said to have taken place at the same hotel where the Secret Service staff were staying in the resort city of Cartagena, and have been confined to barracks.

Obama arrived late Friday in the Caribbean port of Cartagena, which is under security lockdown for the Summit of the Americas, but the allegations against the Secret Service and military completely overshadowed the gathering of regional leaders.

"There have been allegations of misconduct made against Secret Service personnel in Cartagena, Colombia prior to the president's trip," Secret Service special agent in charge Edwin Donovan said in a statement late Friday.

"Because of this, those personnel are being relieved of their assignments, returned to their place of duty, and are being replaced by other Secret Service personnel."

Donovan did not specify the allegations made, but at least one agent had been involved with prostitutes in Cartagena, The Washington Post said, quoting Jon Adler, president of the Federal Law Enforcement Officers Association.

Ronald Kessler, a former Post reporter and author of a book on the Secret Service, told CNN the scandal erupted when one of the agents "did not pay one of the prostitutes, and she complained to the police."

Kessler said 12 agents were being accused of involvement, from interfering with an investigation to actually participating in the alleged misconduct, according to CNN.

Donovan stressed on Friday the Secret Service staffing changes ‘will not affect the comprehensive security plan that has been prepared in advance of the president's trip’.

"This entire matter has been turned over to our Office of Professional Responsibility, which serves as the agency's internal affairs component," he said, without confirming the number of agents sent home.

White House spokesman Jay Carney on Saturday told reporters that Obama had full confidence in the Secret Service.

Shortly after, however, a statement released by the US military's Southern Command, tasked with planning and operations in Central and Southern America, said five of its staff had been linked to the misconduct claims in Cartagena.

Its commander, General Douglas Fraser, was quoted to be ‘disappointed by the entire incident and that this behavior is not in keeping with the professional standards expected of members of the United States military’.

He went on to say that ‘after a thorough investigation, punishment, if appropriate, will take place’, in accordance with military justice.

The revelations came as two small bombs exploded in the capital Bogota, near the US embassy, and another two in Cartagena, though no one was hurt and there was no damage.

The Secret Service, which employs some 3,200 agents and 1,300 uniformed police, has been in the spotlight for a number of notable incidents since Obama took office three years ago.

In late 2009 investigations were launched after an uninvited couple – both aspiring reality TV stars – gatecrashed Obama's first state dinner at the White House, gaining access to the party and even getting photographed with the president.

The couple from Virginia, Tareq and Michaele Salahi, made headlines after attending the early part of the dinner honoring visiting Indian Prime Minister Manmohan Singh despite lacking an invitation or Secret Service clearance.

In November last year an agent was charged with second-degree murder after an incident in Hawaii in November ahead of an APEC summit, The Washington Post reported, while another was charged with drunk driving last August while helping to arrange security for an Obama bus-trip in Iowa.

The Secret Service was created in 1865 to tackle currency counterfeiting, but it gained an expanded role to protect the president in 1901 after the assassination of president William McKinley. Its services were extended to presidential candidates after Robert F. Kennedy was assassinated in 1968.