问题描述
我按照网上的方法在文件log4j.properties中配置#Seteverythingtobeloggedtotheconsolelog4j.rootCategory=WARN,consolelog4j.appender.console=org.apache.log4j.ConsoleAppender这样使用spark-shell,可以看到只有warn信息输出，很简洁。worker.Worker-1-lin-spark.outlin@lin-spark:/opt/data01/spark-1.3.0-bin-2.6.0-cdh5.4.0$bin/spark-shellSparkassemblyhasbeenbuiltwithHive,includingDatanucleusjarsonclasspath16/05/2110:56:52WARNNativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform...usingbuiltin-javaclasseswhereapplicableWelcometo______/__/__________//___\/_/_`/__/'_//___/.__/_,_/_//_/_version1.3.0/_/UsingScalaversion2.10.4(JavaHotSpot(TM)64-BitServerVM,Java1.8.0_05)Typeinexpressionstohavethemevaluated.Type:helpformoreinformation.16/05/2110:56:56WARNUtils:Yourhostname,lin-sparkresolvestoaloopbackaddress:127.0.1.1;using10.170.56.63instead(oninterfaceeth0)16/05/2110:56:56WARNUtils:SetSPARK_LOCAL_IPifyouneedtobindtoanotheraddressSparkcontextavailableassc.SQLcontextavailableassqlContext.但是使用IDEA写完代码后运行，依旧很多INFO，这是怎么回事，怎么处理？6/05/2110:57:52INFOMemoryStore:Blockbroadcast_52_piece0storedasbytesinmemory(estimatedsize2.0KB,free253.4MB)16/05/2110:57:52INFOBlockManagerInfo:Addedbroadcast_52_piece0inmemoryonlocalhost:56191(size:2.0KB,free:256.8MB)16/05/2110:57:52INFOBlockManagerMaster:Updatedinfoofblockbroadcast_52_piece016/05/2110:57:52INFOSparkContext:Createdbroadcast52frombroadcastatDAGScheduler.scala:83916/05/2110:57:52INFODAGScheduler:Submitting1missingtasksfromStage39(MapPartitionsRDD[98]atmapathomework3.scala:67)16/05/2110:57:52INFOTaskSchedulerImpl:Addingtaskset39.0with1tasks16/05/2110:57:52INFOTaskSetManager:Startingtask0.0instage39.0(TID654,localhost,PROCESS_LOCAL,1322bytes)16/05/2110:57:52INFOExecutor:Runningtask0.0instage39.0(TID654)16/05/2110:57:52INFOHadoopRDD:Inputsplit:file:/opt/data02/sparkApp/IndexSearch/IRdata/reut2-007_491:0+450316/05/2110:57:52INFOExecutor:Finishedtask0.0instage39.0(TID654).1845bytesresultsenttodriver16/05/2110:57:52INFOTaskSetManager:Finishedtask0.0instage39.0(TID654)in54msonlocalhost(1/1)16/05/2110:57:52INFOTaskSchedulerImpl:RemovedTaskSet39.0,whosetaskshaveallcompleted,frompool16/05/2110:57:52INFODAGScheduler:Stage39(firstathomework3.scala:68)finishedin0.054s16/05/2110:57:52INFODAGScheduler:Job29finished:firstathomework3.scala:68,took0.056794s
解决方案
解决方案二：
补充一下：完整的log4j.properties文件为：#Seteverythingtobeloggedtotheconsolelog4j.rootCategory=WARN,consolelog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.errlog4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/ddHH:mm:ss}%p%c{1}:%m%n#Settingstoquietthirdpartylogsthataretooverboselog4j.logger.org.eclipse.jetty=WARNlog4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERRORlog4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFOlog4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
解决方案三：
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFOlog4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO这里直接改为log4j.logger.org.apache.spark=WARN见$SPARK_HOME/conf/log4j.log4j.properties没有就cplog4j.properties.templatelog4j.properties你那一大堆info输出都是spark的，其实不建议屏蔽。