球在平板上的鲁棒视觉伺服运动控制

球在平板上的鲁棒视觉伺服运动控制

Mechatronics13(2003)723–738Robustvisualservoingformotioncontroloftheballonaplatea,*bJongHyeonPark,YoungJongLeeaSchoolofMechanicalEngineering,HanyangUniversity,Seoul133-791,RepublicofKoreabDepartmentofPrecisionMechanicalEngineering,HanyangUniversity,Seoul133-791,RepublicofKoreaAbstractThispaperpresentsvisualservoingcontrolforaballonaflatplatetotrackitsdesiredtrajectory.Anindustrialrobotmanipulatorisusedastheactuatorstochangetheangleoftheplatesothattheballcantrackitsgiventrajectoryontheplate.Thecenterpositionoftheballismeasuredwithamachinevisionsystem.BasedontheEulerestimator,analgorithmtofindthecenteroftheballwithinterlaced-scannedimagedataisproposed.Tocopewiththevariationsinthesurfacecharacteristicsoftheplate,andthesizeandmassoftheball,asliding-modecontrolisusedforthevisualservoing.Experimentalresultsshowthattheproposedalgorithmandthesliding-modecontrollerworkwellwithrobustness.Ó2002ElsevierScienceLtd.Allrightsreserved.Keywords:Machinevision;Sliding-modecontrol;Visualservoing;Interlacedscanning1.IntroductionWhenahumanoperatorperformsatask,heorsheusesvarioussensingorganstodealwithanychangeoruncertaintyinvolvedinthetaskandtheenvironment.Amongmanysensingorgans,eyevisionistheprimeinformationroutetogatherthedataontheenvironmentinmanytasks.Theoperator’shand-eyecoordinationisakeytosuccessinmanycases.Forrobots,whichhavethecapabilityofpositioningtheirend-effectorsveryprecisely,withoutsensorssuchasvisionsystems,theyarelimitedtoperformingsimpletasksinapre-determinedenvironment.Asinhumancounterparts,oneoftheprimesensorsthatrobotsneedinperformingtasksdex-terouslyisvision.Visualinformationbecomesmoreimportantespeciallywhenthe*Correspondingauthor.Tel.:+822-2290-0435;fax:+822-2298-4634.E-mailaddress:jongpark@hanyang.ac.kr(JongH.Park).0957-4158/03/$-seefrontmatterÓ2002ElsevierScienceLtd.Allrightsreserved.doi:10.1016/S0957-4158(02)00039-9\n724J.H.Park,Y.J.Lee/Mechatronics13(2003)723–738environmentchangesdynamicallyandnoothernon-contactmeanstomeasurethestatusofthechangingenvironmentisfeasible.Manyresearchershaveusedmachinevisionsystemsinperformingdifficulttasksefficiently.ShiraiandInoue[1]describedhowavisualfeedbackloopcanbeusedtocorrectthepositionofarobottoincreasetaskaccuracy.Hashimoto[2]provedtheusefulnessofthevisualfeedbackusingaroboticsystem.Hutchinsonetal.[3]providesagoodconceptualintroductiontovisualservoingandcategorizesvisualservoingintodifferentsub-categories.Camerascanbefixedormovedaroundin-dependentoftherobotconfiguration,orcanbeattachedtotherobotsothatitspositionandorientationsaredirectlydependentontherobotconfigurationasin[4].Controlsystemscanbeeitherposition-based[5,6]orimage-based[7].Intheformer,thevisualdataisusedtocomputetherelativepositionoftheobject(s)tothecamera.Inthispaper,amachinevisionsystembasedonpositioncontrolisproposedandusedinordertocontrolthetrajectoryoftheballonaflatplate.Thevisionsystem,fixedinalocation,collectstheinformationonthepositionoftheballontheplate.AfastalgorithmtodetectthepositionoftheballontheimageplanecombinedwiththeEulerpositionestimatorresultsinafastcontrol-loopattheregularframerate.A6-DOFindustrialrobotisusedtocontroltheorientationoftheplatetomovetheballaroundontheplate.Thevisualservoingsystemiscontrolledbythesliding-modecontroller,whichhasrobustnesstosystemuncertainty,andunmodelleddistur-bances.Section2describesanalgorithmofsearchingtheballfromtheimagedataandtherelationshipbetweentheworldcoordinatesandtheimagecoordinates.InSection3,thedynamicsofaballonaplateandthesliding-modecontrolleriscovered.ExperimentsandtheirresultsarecoveredinSection4,followedbyconclusionsinSection5.2.Visionsystem2.1.ImageprocessingThevisionsystemusedinthispapertakesbinarydataafterdigitizingtheimagedataatthehardwarelevel,thethresholdlevelsforwhicharestoredinalookuptableoftheframegrabber.Itisadvantageoustousebinarydataindifferentiatinganobjectfromitsnear-monotonicsurroundingsfast[8].Inthevisionsystemused,onefull-frameofimageconsistsof640480pixels.Searchingthewholevideodataofafull-frameforasmallmovingobjectusuallytakesquitelongtime,andthusdegradestheperformanceofthevisualservoingsystem.Thus,basedonthesizeoftheballintheimageplane,only100100-pixel-sizedimagedataaroundtheestimatedpositionoftheballaresearchedtofindtheprecisepositionoftheball(center),asshowninFig.1.Estimatedpositionoftheballatðkþ1Þ-thframe,denotedbyxbðkþ1Þ,iscalculatedusingtheEulerestimator[9],i.e.,\nJ.H.Park,Y.J.Lee/Mechatronics13(2003)723–738725Fig.1.Estimationoftheballcenter.ixðkÞixðk1Þix^ibbbðkþ1Þ¼xbðkÞþ;ð1ÞDtwhereixðkÞisthemeasuredpositionoftheballontheimageplaneatkthframeandbDtisthetimegapbetweenthekthandtheðk1Þthframesamples.Thecenterpositionoftheballintheimageplaneiscomputedbyaveragingthemaximumpositionandtheminimumpositionsoftheimageblobassociatedwiththeball.Thus,ixðkÞ¼1½þmaxix2BðkÞminix2BðkÞ;ð2Þb2whereix2R2denotesthepixelindexontheimageplaneandBðkÞdenotestheimageblobatframekthatrepresentstheball.Whenanobjectmovesfast,itsimagetakenfromvideocameraswithinterlacedscanningmaybecomeblurred.Fig.2showstheimagestakenwhenacircularobjectmovesfasthorizontallyandvertically.Thus,interlacedscanning,whichscansevenandoddfieldsofthevideosignalseparately,introducesimagedistortions[10],andthuspreventsanaccuratemeasurementofthepositionoftheobject.Tominimizetheerrorinthepositionoftheball,onlyonefield(evenorodd)ofthevideodataisusedtomeasurethecenteroftheball.Fig.3showstheprioritymasksusedfortheal-gorithm.MaskMxmininthefigureisusedtofindthepixelthatrepresentsthelower-boundoftheimagebloboftheballinthex-direction.Tofindthebound,thefollowingstepsaretaken:Step1.Placethecenterofthemask,mcatthepreviouslyestimatedcenteroftheball,ix^ðk1Þ.b\n726J.H.Park,Y.J.Lee/Mechatronics13(2003)723–738Fig.2.Imagedistortionduetotheinterlacedscanning:whenacircularobjectmoves:(a)horizontally,and(b)vertically.Step2.Ifthepixelat‘‘first’’isoneofthepixelsrepresentingtheball,thenthemaskismovedsuchthatitscenterislocatedatthatpixel.Then,gobacktoStep1.Step3.Ifthepixelat‘‘first’’doesnotrepresenttheball,checkthepixelat‘‘second’’ofthemask.Ifitrepresentstheball,thenthecenterofthemaskismovedtothepositionofthepixel.Then,gobacktoStep1.Step4.Ifthepixelat‘‘second’’isnotassociatedwiththeimageoftheball,checkifthepixelat‘‘third’’isassociatedwiththeimageoftheball.Ifso,thecenterofthemaskisshifteddowntothepositionofthepixel,andthengobacktoStep1.Otherwise,thelower-boundinthex-direction,i.e.,minix2BðkÞ,issetasthepixelatthecenterofthemask.Tofindtheupper-boundpixelinthex-direction,Mxmax,isusedinasimilarfashiontomovethemasktotherightoftheobjectblobontheimageplane.Sim-\nJ.H.Park,Y.J.Lee/Mechatronics13(2003)723–738727Fig.3.Amaskusedincomputingthecenterofaball.ilarly,tofindtheupper-andlower-boundsinthey-direction,masksMymaxandMyminareused,respectively.2.2.CameraprojectionmodelInsettinguptherelationshipbetweenanobjectanditsimageontheimageplane,theso-calledscaledorthographicprojectionisoftenusedespeciallywhenthemotionrangeoftheobjectinthedirectionperpendiculartotheimageplaneissmallcom-paredwiththedistancebetweentheobjectandthecamera.Sincethatthedepthvariationoftheballontheplateissmallduetosmallchangesintheplaneorien-tationasshowninFig.4,thescaledorthographicprojectionisused.Therelationshipbetweenthereferencecoordinatesandtheimagecoordinatesinthescaledortho-graphicprojection[11]canbeexpressedbyixxi¼Sc;ð3Þyywhereðx;yÞisthepositionoftheobjectaboutthecamerareferencecoordinateandðix;iyÞisthepositionoftheobjectintheimagecoordinatesontheimageplane.And,\n728J.H.Park,Y.J.Lee/Mechatronics13(2003)723–738Fig.4.Configurationofthecameraandtheplate.22matrixSc2Risascalefactor,whichdependsonmanyfactorsincludingthedistancebetweenthecameraandtheobject,andthefocallengthofthecameralens.3.Controllerdesign3.1.EquationsofmotionfortheballonaplateSupposethattheballandtheplatearealignedasinFig.5.Thex–yplaneiscoplanarwiththeplaneonwhichtheballislocated.Thez-directionisupwardandnormaltotheplane.Itwillbeassumedthattheballontheplaneisperfectlyasphere.Then,fromNewton’ssecondlaw,TymR€xJayþmgRsinhy¼0;ð4ÞTxþmR€yJaxþmgRsinhx¼0;ð5ÞwhereR,mandJaretheradius,themassandthemomentinertiaoftheball;hxandhyaretheinclinationangleoftheplateinthex-andy-direction;axandayaretheangularaccelerationoftheballinthex-andy-direction;TxandTyarethetorquegeneratedinthex-andy-directionrespectively,atthecontactareabetweentheballandtheplate,associatedwiththedeformationoftheplate(andtheball)duetothecontactforcebetweenthem.TorqueTxandTyarestronglydependsonthetypeofthematerialsandthesurfacecondition,suchasroughness,oftheballandtheplate.Iftheballisuniform,itsmomentofinertiaisJ¼2mR2:5\nJ.H.Park,Y.J.Lee/Mechatronics13(2003)723–738729Fig.5.Theballrollsdowntheinclinedplate.Fig.6.Overallsystemblockdiagram.\n730J.H.Park,Y.J.Lee/Mechatronics13(2003)723–738Ingeneral,foraballthatismadeoftwodifferentmaterials,22Ml2MhJ¼kmR0;Fig.10.Errorsintrackinganellipticaltrajectorywithacompoundballandaplasticplate.\n734J.H.Park,Y.J.Lee/Mechatronics13(2003)723–738matrixKmustbeselectedsuchthatk1111PðIQQQbÞð€xdK~x_f^ÞþQQQbðff^ÞþQQQbg:ð16Þk2UsingtheboundsoffandQinEqs.(10)and(13),theconstraintonKmaybealsoexpressedbyk11=21=21=21=2kPQUQL€xdK~x_f^þQUQLðDfþgÞ:ð17Þ2ByusingthesaturationfunctionwithboundarylayerthicknessUinsteadofthesignfunctioninordertoremovechatteringfromthecontrollaw,thesliding-modecontrolbecomeshi1suðtÞ¼Qb€xdðtÞKðx_ðtÞx_dðtÞÞf^Ksat:ð18ÞU4.ExperimentsandresultsAseriesofexperimentswereperformedtomeasuretheperformanceofthevisualservoingsystem.TheoverallsystemcomponentsusedintheexperimentsareshowninFig.6.Thehostcomputer,aPentiumII,generatesthedesiredtrajectoryfortheballandalsoworksasthesliding-modecontroller.Thevisionboard,aframegrabber,collectsandstoresvideodatainthebinaryforminthehostcomputermemory.TheMMCboardinaPCslotgeneratestheservocontrolinputtotheservomotordrivesforan6-DOFindustrialrobot,SamsungFara-AS1.Theservomotordrivesareoff-the-shelfproducts.ThedesiredtrajectoryfortheballtotrackontheFig.11.Trackinganellipticaltrajectorywithasteelballandarubber-coatedplate.\nJ.H.Park,Y.J.Lee/Mechatronics13(2003)723–738735platearegivenintheformofanellipsoidandatriangle.Inordertoverifytherobustnessofthesystem,differentplatesanddifferentballsweretriedout.Inthefirstexperiment,theballontheplateistofollowanellipticaltrackataconstantspeed.Theballis17mmindiameter,madeofsteel,andweighsabout28g.Theplateismadeofplasticandhasasmoothflatsurface.TheresultingtrackoftheballisshowninFig.7intermsoftheimagecoordinates.Adashedlineistheref-erencetrajectoryandasolidlineisthetrajectoryoftheball.Marks‘þ’and‘’denotetheinitialandthefinalpositionoftheball,respectively.Fig.8showsthetrackingerrorsinthex-andy-axis,whicharesmallerthan7mm,correspondingto10pixels.Inthenextexperiment,thesameplatewasusedbutwithadifferentball,acompoundballmadeofhollowplasticandrubber.Thetrackingaccuracygotworsethanthatinthefirstexperiment,asshowninFig.9.Itisbecausetheinertiaofballwasdifferentfrommodel.Fig.10indicatesthatthemaximumtrackingerrorineachdirectionisabout13mm,correspondingto20pixels.Fig.12.Errorsintrackinganellipticaltrajectorywithasteelballandarubber-coatedplate.\n736J.H.Park,Y.J.Lee/Mechatronics13(2003)723–738Fig.13.Trackingatriangulartrajectorywithasteelballandaplasticplate.Inthethirdexperiment,weusedthesameballastheoneusedinthefirstex-periment,butadifferentplate,whichwascoatedwithrubber.TheballtrackedthedesiredtrajectorywellasshowninFig.11.However,theaccuracyintrackingwasworsethanthoseintheprevioustwoexperiments.Itisbecausethereexistdampingeffectsontheplateduetothedeformationoftheplatedrubber.Fig.12showsthatthemaximumtrackingerrorisabout16.7mm,correspondingto25pixels.Next,atriangulartrajectorywasusedinsteadoftheellipticalone,butunderthesameconditionastheoneinthefirstexperiment.Theformerresultedinatrajectorywithsuddensurgesanddropsinthetrajectorywhilethelatterresultedinasmoothsinusoidaltrajectoryineachdirectionofthecoordinates.Fig.13showsthattheballfollowsthedesiredtrajectoryquitewell.TheerrorplotineachdirectioninFig.14showsthatthemagnitudeofthetrackingerrorwaslessthan12pixels.5.ConclusionsInthispaper,weproposedavisualservoingsystemthatconsistsofamachinevisionsystemandanindustrialrobot,tomovetheflatplateheldbytherobotsothattheballontheplatecantrackitsdesiredtrajectory.Inordertoimprovethetrackingperformance,weusedthesearchareareductionmethodwhichguaranteehighsamplingfrequency,andpresentedthecenterdetectionalgorithmthatovercomestheinterlacedscanningeffect.Tocopewiththeuncertaintyofthesystemmodelin-cludingthecontactbetweentheballandtheplate,materialsoftheballandtheplate,andthesizeandweightoftheball,asliding-modecontrollerwasdesigned.Through\nJ.H.Park,Y.J.Lee/Mechatronics13(2003)723–738737Fig.14.Errorsintrackingatriangulartrajectorywithasteelballandaplasticplate.aseriesofexperiments,itwasshownthatthesystemperformedtheballtrackingverywellandthattheperformancewasrobusttotheuncertainty.References[1]ShiraiY,InoueH.Guidingarobotbyvisualfeedbackinassemblingtasks.PatternRecognit1973;5:99–108.[2]HashimotoK,NoritsuguT.Modelingandcontrolofroboticyoyowithvisualfeedback.In:ProceedingsofIEEEInternationalConferenceonRoboticsandAutomation,Minneapolis,USA.1996.p.2650–5.[3]HutchinsonS,HagerGD,CorkePI.Atutorialonvisualservocontrol.IEEETransRobotAutomat1996;12(5):651–70.[4]KellyR,CarelliR,NasisiO,KuchenB,ReyesF.Stablevisualservoingofcamera-in-handroboticsystems.IEEE/ASMETransMechatron2000;5(1):39–48.[5]WilliamJ,HullsCCW,BellGS.Relativeend-effectorcontrolusingCartesianpositionbasedvisualservoing.IEEETransRobotAutomat1996;12(5):684–96.\n738J.H.Park,Y.J.Lee/Mechatronics13(2003)723–738[6]AllottaB,ColomboC.Ontheuseoflinearcamera–objectinteractionmodelsinvisualservoing.IEEETransRobotAutomat1999;15(2):350–7.[7]BorsAG,PitasI.Predictionandtrackingofmovingobjectsinimagesequences.IEEETransImageProcess2000;9(8):1441–5.[8]GonzalezRC,WoodsRE.Digitalimageprocessing.Reading,MA:Addison-Wesley;1992.[9]PiepmeierJA.Trackingamovingtargetwithmodelindependentvisualservoing:apredictiveestimationapproach.In:ProceedingsofIEEEInternationalConferenceonRoboticsandAutoma-tion,Leuven,Belgium.1998.p.2652–7.[10]YokokohjiY.Accurateimageoverlayonhead-mounteddisplaysusingvisionandaccelerometers.In:ProceedingsofIEEEInternationalConferenceonRoboticsandAutomation,Detroit,USA.1999.p.3243–8.[11]HagerGD,CorkePI.Atutorialonvisualservocontrol.IEEETransRobotAutomat1996;12:651–70.[12]SlotineJ-JE,LiW.Appliednonlinearcontrol.EnglewoodCliffs,NJ:Prentice-Hall;1991.[13]EdwardsC,SpurgeonSK.Slidingmodecontrol:theoryandapplications.London:Taylor&Francis;1998.[14]KellyR.Robustasymptoticallystablevisualservoingofplanarrobots.IEEETransRobotAutomat1996:759–66.[15]MalisE,ChaumetteF,BoudetS.21/2visualservoing.IEEETransRobotAutomat1999;15(2):238–50.[16]NguyenM-C,GraefeV.Self-learningvision-guidedrobotsforsearchingandgraspingobjects.In:ProceedingsofRoboticsandAutomation.2000.p.1633–8.