使用hadoop restful api实现对集群信息的统计

（适用于hadoop 2.7及以上版本）

涉及到RESTful API

1. 统计HDFS文件系统实时使用情况

  • URL
    http://emr-header-1:50070/webhdfs/v1/?user.name=hadoop&op=GETCONTENTSUMMARY
  • 返回结果： 
    {
"ContentSummary":
{
"directoryCount": 2,
"fileCount"     : 1,
"length"        : 24930,
"quota"         : -1,
"spaceConsumed" : 24930,
"spaceQuota"    : -1
}
}
  • 关于返回结果的说明： 
    {
"name"      : "ContentSummary",
"properties":
{
"ContentSummary":
{
  "type"      : "object",
  "properties":
  {
    "directoryCount":
    {
      "description": "The number of directories.",
      "type"       : "integer",
      "required"   : true
    },
    "fileCount":
    {
      "description": "The number of files.",
      "type"       : "integer",
      "required"   : true
    },
    "length":
    {
      "description": "The number of bytes used by the content.",
      "type"       : "integer",
      "required"   : true
    },
    "quota":
    {
      "description": "The namespace quota of this directory.",
      "type"       : "integer",
      "required"   : true
    },
    "spaceConsumed":
    {
      "description": "The disk space consumed by the content.",
      "type"       : "integer",
      "required"   : true
    },
    "spaceQuota":
    {
      "description": "The disk space quota.",
      "type"       : "integer",
      "required"   : true
    }
  }
}
}
}
  • 注意length与spaceConsumed的关系，跟hdfs副本数有关。
  • 如果要统计各个组工作目录的使用情况，使用如下请求：
    http://emr-header-1:50070/webhdfs/v1/user/feed_aliyun?user.name=hadoop&op=GETCONTENTSUMMARY

2. 查看集群的实时信息和状态

  • URL

http://emr-header-1:8088/ws/v1/cluster

  • 返回结果
{
    "clusterInfo": {
        "id": 1495123166259,
        "startedOn": 1495123166259,
        "state": "STARTED",
        "haState": "ACTIVE",
        "rmStateStoreName": "org.apache.hadoop.yarn.server.resourcemanager.recovery.NullRMStateStore",
        "resourceManagerVersion": "2.7.2",
        "resourceManagerBuildVersion": "2.7.2 from 4bee04d3d1c27d7ef559365d3bdd2a8620807bfc by root source checksum c63f7cc71b8f63249e35126f0f7492d",
        "resourceManagerVersionBuiltOn": "2017-04-17T12:28Z",
        "hadoopVersion": "2.7.2",
        "hadoopBuildVersion": "2.7.2 from 4bee04d3d1c27d7ef559365d3bdd2a8620807bfc by root source checksum 3329b146070a2bc9e249fa9ba9fb55",
        "hadoopVersionBuiltOn": "2017-04-17T12:18Z",
        "haZooKeeperConnectionState": "ResourceManager HA is not enabled."
    }
}

3. 查看资源队列的实时信息，包括队列的配额信息、资源使用实时情况

  • URL

http://emr-header-1:8088/ws/v1/cluster/scheduler

  • 返回结果
{
    "scheduler": {
        "schedulerInfo": {
            "type": "capacityScheduler",
            "capacity": 100,
            "usedCapacity": 0,
            "maxCapacity": 100,
            "queueName": "root",
            "queues": {
                "queue": [
                    {
                        "type": "capacitySchedulerLeafQueueInfo",
                        "capacity": 1,
                        "usedCapacity": 0,
                        "maxCapacity": 90,
                        "absoluteCapacity": 1,
                        "absoluteMaxCapacity": 90,
                        "absoluteUsedCapacity": 0,
                        "numApplications": 0,
                        "queueName": "algorithm_aliyun",
                        "state": "RUNNING",
                        "resourcesUsed": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "hideReservationQueues": false,
                        "nodeLabels": [
                            "*"
                        ],
                        "numActiveApplications": 0,
                        "numPendingApplications": 0,
                        "numContainers": 0,
                        "maxApplications": 100,
                        "maxApplicationsPerUser": 100,
                        "userLimit": 100,
                        "users": null,
                        "userLimitFactor": 1,
                        "AMResourceLimit": {
                            "memory": 11776,
                            "vCores": 7
                        },
                        "usedAMResource": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "userAMResourceLimit": {
                            "memory": 160,
                            "vCores": 1
                        },
                        "preemptionDisabled": true
                    },
                    {
                        "type": "capacitySchedulerLeafQueueInfo",
                        "capacity": 1,
                        "usedCapacity": 0,
                        "maxCapacity": 90,
                        "absoluteCapacity": 1,
                        "absoluteMaxCapacity": 90,
                        "absoluteUsedCapacity": 0,
                        "numApplications": 0,
                        "queueName": "dcps_aliyun",
                        "state": "RUNNING",
                        "resourcesUsed": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "hideReservationQueues": false,
                        "nodeLabels": [
                            "*"
                        ],
                        "numActiveApplications": 0,
                        "numPendingApplications": 0,
                        "numContainers": 0,
                        "maxApplications": 100,
                        "maxApplicationsPerUser": 100,
                        "userLimit": 100,
                        "users": null,
                        "userLimitFactor": 1,
                        "AMResourceLimit": {
                            "memory": 11776,
                            "vCores": 7
                        },
                        "usedAMResource": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "userAMResourceLimit": {
                            "memory": 160,
                            "vCores": 1
                        },
                        "preemptionDisabled": true
                    },
                    {
                        "type": "capacitySchedulerLeafQueueInfo",
                        "capacity": 31,
                        "usedCapacity": 0,
                        "maxCapacity": 100,
                        "absoluteCapacity": 31,
                        "absoluteMaxCapacity": 100,
                        "absoluteUsedCapacity": 0,
                        "numApplications": 0,
                        "queueName": "default",
                        "state": "RUNNING",
                        "resourcesUsed": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "hideReservationQueues": false,
                        "nodeLabels": [
                            "*"
                        ],
                        "numActiveApplications": 0,
                        "numPendingApplications": 0,
                        "numContainers": 0,
                        "maxApplications": 3100,
                        "maxApplicationsPerUser": 3100,
                        "userLimit": 100,
                        "users": null,
                        "userLimitFactor": 1,
                        "AMResourceLimit": {
                            "memory": 13088,
                            "vCores": 8
                        },
                        "usedAMResource": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "userAMResourceLimit": {
                            "memory": 4064,
                            "vCores": 3
                        },
                        "preemptionDisabled": true
                    },
                    {
                        "type": "capacitySchedulerLeafQueueInfo",
                        "capacity": 15.000001,
                        "usedCapacity": 0,
                        "maxCapacity": 100,
                        "absoluteCapacity": 15.000001,
                        "absoluteMaxCapacity": 100,
                        "absoluteUsedCapacity": 0,
                        "numApplications": 0,
                        "queueName": "feed_aliyun",
                        "state": "RUNNING",
                        "resourcesUsed": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "hideReservationQueues": false,
                        "nodeLabels": [
                            "*"
                        ],
                        "numActiveApplications": 0,
                        "numPendingApplications": 0,
                        "numContainers": 0,
                        "maxApplications": 1500,
                        "maxApplicationsPerUser": 7500,
                        "userLimit": 100,
                        "users": null,
                        "userLimitFactor": 5,
                        "AMResourceLimit": {
                            "memory": 12320,
                            "vCores": 8
                        },
                        "usedAMResource": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "userAMResourceLimit": {
                            "memory": 9856,
                            "vCores": 7
                        },
                        "preemptionDisabled": true
                    },
                    {
                        "type": "capacitySchedulerLeafQueueInfo",
                        "capacity": 51,
                        "usedCapacity": 0,
                        "maxCapacity": 90,
                        "absoluteCapacity": 51,
                        "absoluteMaxCapacity": 90,
                        "absoluteUsedCapacity": 0,
                        "numApplications": 0,
                        "queueName": "hot_aliyun",
                        "state": "RUNNING",
                        "resourcesUsed": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "hideReservationQueues": false,
                        "nodeLabels": [
                            "*"
                        ],
                        "numActiveApplications": 0,
                        "numPendingApplications": 0,
                        "numContainers": 0,
                        "maxApplications": 5100,
                        "maxApplicationsPerUser": 5100,
                        "userLimit": 100,
                        "users": null,
                        "userLimitFactor": 1,
                        "AMResourceLimit": {
                            "memory": 11776,
                            "vCores": 7
                        },
                        "usedAMResource": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "userAMResourceLimit": {
                            "memory": 6688,
                            "vCores": 5
                        },
                        "preemptionDisabled": true
                    },
                    {
                        "type": "capacitySchedulerLeafQueueInfo",
                        "capacity": 1,
                        "usedCapacity": 0,
                        "maxCapacity": 90,
                        "absoluteCapacity": 1,
                        "absoluteMaxCapacity": 90,
                        "absoluteUsedCapacity": 0,
                        "numApplications": 0,
                        "queueName": "push_aliyun",
                        "state": "RUNNING",
                        "resourcesUsed": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "hideReservationQueues": false,
                        "nodeLabels": [
                            "*"
                        ],
                        "numActiveApplications": 0,
                        "numPendingApplications": 0,
                        "numContainers": 0,
                        "maxApplications": 100,
                        "maxApplicationsPerUser": 100,
                        "userLimit": 100,
                        "users": null,
                        "userLimitFactor": 1,
                        "AMResourceLimit": {
                            "memory": 11776,
                            "vCores": 7
                        },
                        "usedAMResource": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "userAMResourceLimit": {
                            "memory": 160,
                            "vCores": 1
                        },
                        "preemptionDisabled": true
                    }
                ]
            }
        }
    }
}

4. 查看实时的作业列表，列表信息中也包含了作业运行的详情信息，包括作业名称、id、运行状态、起止时间，资源使用情况。

  • URL

http://emr-header-1:8088/ws/v1/cluster/apps

  • 返回结果
{
  "apps":
  {
    "app":
    [
       {
          "finishedTime" : 1326815598530,
          "amContainerLogs" : "http://host.domain.com:8042/node/containerlogs/container_1326815542473_0001_01_000001",
          "trackingUI" : "History",
          "state" : "FINISHED",
          "user" : "user1",
          "id" : "application_1326815542473_0001",
          "clusterId" : 1326815542473,
          "finalStatus" : "SUCCEEDED",
          "amHostHttpAddress" : "host.domain.com:8042",
          "progress" : 100,
          "name" : "word count",
          "startedTime" : 1326815573334,
          "elapsedTime" : 25196,
          "diagnostics" : "",
          "trackingUrl" : "http://host.domain.com:8088/proxy/application_1326815542473_0001/jobhistory/job/job_1326815542473_1_1",
          "queue" : "default",
          "allocatedMB" : 0,
          "allocatedVCores" : 0,
          "runningContainers" : 0,
          "memorySeconds" : 151730,
          "vcoreSeconds" : 103
       },
       {
          "finishedTime" : 1326815789546,
          "amContainerLogs" : "http://host.domain.com:8042/node/containerlogs/container_1326815542473_0002_01_000001",
          "trackingUI" : "History",
          "state" : "FINISHED",
          "user" : "user1",
          "id" : "application_1326815542473_0002",
          "clusterId" : 1326815542473,
          "finalStatus" : "SUCCEEDED",
          "amHostHttpAddress" : "host.domain.com:8042",
          "progress" : 100,
          "name" : "Sleep job",
          "startedTime" : 1326815641380,
          "elapsedTime" : 148166,
          "diagnostics" : "",
          "trackingUrl" : "http://host.domain.com:8088/proxy/application_1326815542473_0002/jobhistory/job/job_1326815542473_2_2",
          "queue" : "default",
          "allocatedMB" : 0,
          "allocatedVCores" : 0,
          "runningContainers" : 1,
          "memorySeconds" : 640064,
          "vcoreSeconds" : 442
       }
    ]
  }
}
  • 如果要统计固定时间段的，可以加上"?finishedTimeBegin={时间戳}&finishedTimeEnd={时间戳}"参数，例如
    http://emr-header-1:8088/ws/v1/cluster/apps?finishedTimeBegin=1496742124000&finishedTimeEnd=1496742134000

5. 统计作业扫描的数据量情况

job扫描的数据量，需要通过History Server的RESTful API查询，MapReduce的和Spark的又有一些差异。

5.1 Mapreduce job扫描数据量

  • URL

http://emr-header-1:19888/ws/v1/history/mapreduce/jobs/job_1495123166259_0962/counters

  • 返回结果
{
   "jobCounters" : {
      "id" : "job_1326381300833_2_2",
      "counterGroup" : [
         {
            "counterGroupName" : "Shuffle Errors",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "BAD_ID"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "CONNECTION"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "IO_ERROR"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "WRONG_LENGTH"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "WRONG_MAP"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "WRONG_REDUCE"
               }
            ]
          },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2483,
                  "name" : "FILE_BYTES_READ"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 108525,
                  "name" : "FILE_BYTES_WRITTEN"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "FILE_READ_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "FILE_LARGE_READ_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "FILE_WRITE_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 48,
                  "name" : "HDFS_BYTES_READ"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "HDFS_BYTES_WRITTEN"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1,
                  "name" : "HDFS_READ_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "HDFS_LARGE_READ_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "HDFS_WRITE_OPS"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1,
                  "name" : "MAP_INPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1200,
                  "name" : "MAP_OUTPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 4800,
                  "name" : "MAP_OUTPUT_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2235,
                  "name" : "MAP_OUTPUT_MATERIALIZED_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 48,
                  "name" : "SPLIT_RAW_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "COMBINE_INPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "COMBINE_OUTPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1200,
                  "name" : "REDUCE_INPUT_GROUPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2235,
                  "name" : "REDUCE_SHUFFLE_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1200,
                  "name" : "REDUCE_INPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "REDUCE_OUTPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2400,
                  "name" : "SPILLED_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1,
                  "name" : "SHUFFLED_MAPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "FAILED_SHUFFLE"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1,
                  "name" : "MERGED_MAP_OUTPUTS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 113,
                  "name" : "GC_TIME_MILLIS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1830,
                  "name" : "CPU_MILLISECONDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 478068736,
                  "name" : "PHYSICAL_MEMORY_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2159284224,
                  "name" : "VIRTUAL_MEMORY_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 378863616,
                  "name" : "COMMITTED_HEAP_BYTES"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "BYTES_READ"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "BYTES_WRITTEN"
               }
            ]
         }
      ]
   }
}

其中org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter里面的BYTES_READ为job扫描的数据量
具体参数：https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html#Job_Counters_API

5.2 Mapreduce job扫描数据量

  • URL

http://emr-header-1:18080/api/v1/applications/application_1495123166259_1050/executors

每个executor的totalInputBytes总和为整个job的数据扫描量。
更多参考：http://spark.apache.org/docs/latest/monitoring.html

时间： 2024-10-17 10:20:05

使用hadoop restful api实现对集群信息的统计的相关文章

日志-求救啊！Hadoop 2.2.0 搭建集群 启动hdfs时候 namenode 启动后报空指针

问题描述 求救啊!Hadoop 2.2.0 搭建集群 启动hdfs时候 namenode 启动后报空指针 日志如下: 2015-02-07 01:01:46,610 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Error encountered requiring NN shutdown. Shutting down immediately. java.lang.NullPointerException at org.apache

【干货】Apache Hadoop 2.8 完全分布式集群搭建超详细过程，实现NameNode HA、ResourceManager HA高可靠性

最近在自己的笔记本电脑上搭建了Apache Hadoop分布式集群,采用了最新的稳定版本2.8,并配置了NameNode.ResourceManager的HA高可用,方便日常对Hadoop的研究与测试工作.详细的搭建过程如下: 1.安装docker,创建docker容器,用于搭建hadoop节点 docker真是个好东西啊,当要在自己的笔记本上搭建分布式集群时,由于CPU.内存.磁盘有限,无法在VMware上虚拟出太多节点,这时使用docker创建几个容器,就能轻松搭建一个分布式集群了. (1)

Hadoop 2.5.2分布式集群配置

这里一共有三台机器,系统为Ubuntu 14.04.2,其中一台为master 其余两台为slave 1.集群之间各台机器上添加相同的用户 首先用adduser命令添加一个普通用户,命令如下: #adduser lq //添加一个名为tommy的用户 #passwd lq //修改密码 Changing password for user lq. New UNIX password: //在这里输入新密码 Retype new UNIX password: //再次输入新密码 passwd: a

Hadoop YARN学习之监控集群监控Nagios（4）

doop YARN学习之监控集群监控Nagios(4) 1. Nagios是一个流行的开源监控工具,可以用来监控Hadoop集群. 2. 监控基本的Hadoop服务 调试好脚本后命名为chek_resource_manager.sh,并把它放在Nagios的插件目录中. 加载Nagios插件向hadoop-cluster.cfg添加如下信息 define command{ command_name check_resource_manager command_line /usr/lib64/na

Hadoop学习之HBase的集群环境搭建

HBase的集群环境搭建 该集成环境是在伪分布搭建的基础上搭建 1.   修改原来的伪分布hadoop1上的hbase的配置文件 #cd /usr/local/hbase/conf/ 待修改的文件:hbase-env.sh.hbase-site.xml.regionservers #vim hbase-env.sh 使用搭建的zookeeper集群环境,因此hbase自带的zookeeper设置为false,不启动. #vim hbase-site.xml 将zookeeper集群所在的主机名,

Hadoop单机版和全分布式(集群)安装_linux shell

Hadoop,分布式的大数据存储和计算, 免费开源!有Linux基础的同学安装起来比较顺风顺水,写几个配置文件就可以启动了,本人菜鸟,所以写的比较详细.为了方便,本人使用三台的虚拟机系统是Ubuntu-12.设置虚拟机的网络连接使用桥接方式,这样在一个局域网方便调试.单机和集群安装相差不多,先说单机然后补充集群的几点配置. 第一步,先安装工具软件编辑器:vim 复制代码 代码如下: sudo apt-get install vim ssh服务器: openssh,先安装ssh是为了使用远程终端工

Hadoop集群容易被攻击的几个场景

本文讲的是Hadoop集群容易被攻击的几个场景, 前言 Hadoop是成为大数据分析平台的主流产品,其安全性一直深受诟病,而针对安全的各种配置也都比较复杂,因此我们在一些场景的配置下进行安全测试,看看究竟哪些场景可能会被攻击. 1.HDFS脆弱性测试 1．1.用户冒充,访问任意文件操作 受影响的环境:在hdfs-core.xml文件中配置,Simple模式,dfs.permissions.enabled开启或关闭均可. 方法:利用客户机1远程进行操作,如果当前客户机1和2分别建立不同密码的用户u

Hadoop集群搭建

目的 本文描述了如何安装.配置和管理有实际意义的Hadoop集群,其规模可从几个节点的小集群到几千个节点的超大集群. 如果你希望在单机上安装Hadoop玩玩,从这里能找到相关细节. 先决条件确保在你集群中的每个节点上都安装了所有必需软件. 获取Hadoop软件包. 安装 安装Hadoop集群通常要将安装软件解压到集群内的所有机器上. 通常,集群里的一台机器被指定为 NameNode,另一台不同的机器被指定为JobTracker.这些机器是masters.余下的机器即作为DataNode也作为Ta

如何从k8s集群外的机器上调用k8s的API(可调用多个k8s的apiserver)

因为公司项目需要,k8s的集群管理由其它人负责,而我们的自动化部署平台需要从k8s集群外的机器上调用k8s主节点的apiserver提供的API. 由于集群是用kubeadm安装的,证书验证齐全,那么,调用API时,只能通过kubectl proxy的方式进行调用管理. 操作步骤如下:   一,在集群外的机器上安装好kubectl的rpm包. rpm -ivh kubectl-1.7.0-0.x86_64.rpm 二,将k8s上在安装过程中生成的config文件重命名,CP到操作用户的家目录的.