This is an old version of this page. You can view the most recent version or browse the history.

environmental_protection_grade

基本信息

环保等级

数据名称（中文）

环保等级

数据英文名称

environmental_protection_grade

采集网站（采集入口）

官网PC端入口:
江苏：https://hblp.jsshbt.cn/shencai-envfacial-web/service/envFacial/hblp/tEnvBasKeylistModel/queryKeyListBusinessNumber
浙江：http://223.4.71.96/portal/data/api/auto
福建：http://220.160.52.213:20071/api/template/page/p_list_eval_credit
四川：http://103.203.219.138:18081/data/w/evaluateResults/list4Public
...

采集文件存放路径：
/data/gravel_spiders/environmental_protection

采集频率及采集策略

存量更新策略

目前全量更新一轮

增量采集策略

爬虫

环保等级爬虫 environmental_protection

责任人

蒋家升

爬虫名称

environmental_protection

代码地址

项目地址: http://tech.pingansec.com/granite/project-gravel/-/tree/develop_environmental_protection_grade/scrapy_spiders

队列名称及队列地址

redis host: redis://:utn@0818@bdp-mq-001.redis.rds.aliyuncs.com:6379/0
redis port: 6379
redis db: 0
redis key:
- environmental_protection

优先级队列说明

environmental_protection 支持队列优先级

任务来源

任务输入参数（样例）

其中CRNo（公司编号）为必需

{'province': 'jiangsu', "step": "start"}
{'province': 'zhejiang', "step": "start"}
{'province': 'fujian', "step": "start"}
{'province': 'sichuan', "step": "start"}

任务样例

任务参数说明

{'province': 'jiangsu', "step": "start", "index": 0}

主要参数

province: 省份拼音

非必要参数

step: 步骤

index: 翻页的页数

data_type说明

list: 列表页数据

福建：
list_of_normal: 全程公示
list_of_red: 红黑榜

...开发中

爬虫结果的超级数据

实际爬虫结果的数据结构

江苏：

{
  "data":
  [
    {
      "creditDataID": "202110242303108479a164c13b45caa51598e28aac5b69",
      "creditLevel": 4,
      "creditName": "一般守信",
      "efResultID": "13574254755210144971",
      "fullName": "宿迁市-宿豫区-宿豫经济开发区",
      "manageLevel": 2,
      "manageName": "二星",
      "spCode": "142332828000",
      "spName": "宿迁市罐头食品有限责任公司",
      "time": "2021-10-25"
    },
    {
      "creditDataID": "202110242305284f9a3b85fc214f13b2cb5944fbec7e36",
      "creditLevel": 4,
      "creditName": "一般守信",
      "efResultID": "13574254755210175105",
      "fullName": "常州市-武进区",
      "manageLevel": 5,
      "manageName": "五星",
      "spCode": "3101120200000099",
      "spName": "江苏绿浥农业科技股份有限公司",
      "time": "2021-10-25"
    },
    {
      "creditDataID": "202110242308191eaf0c8cb8174469bc192039d249540c",
      "creditLevel": 4,
      "creditName": "一般守信",
      "efResultID": "13574254755210175285",
      "fullName": "苏州市-姑苏区",
      "manageLevel": 5,
      "manageName": "五星",
      "spCode": "3200000200000537",
      "spName": "中建三局第一建设工程有限责任公司",
      "time": "2021-10-25"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list",
  "spider_start_time": "2021-10-25 17:17:52.098",
  "spider_end_time": "2021-10-25 17:17:54",
  "task_params":{
    "province": "jiangsu",
    "step": "start",
    "index": 1
  },
  "metadata":{
    "province": "jiangsu",
    "index": 1
  },
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.1.18"
}

浙江：

{
  "data":
  [
    {
      "city": "舟山市",
      "level_title": "优秀",
      "district": "普陀区",
      "level_code": "A",
      "social_credit_code": "91330903336897819J",
      "score_code": "8ca03c8e-634c-49cb-b35c-bac95ce46910",
      "ent_name": "舟山丰瑞海洋生物制品有限公司",
      "region_code": "330903",
      "release_time": 1635264000000
    },
    {
      "city": "舟山市",
      "level_title": "优秀",
      "district": "普陀区",
      "level_code": "A",
      "social_credit_code": "913309031487170831",
      "score_code": "8ca03c8e-634c-49cb-b35c-bac95ce46910",
      "ent_name": "中石化浙江舟山石油有限公司",
      "region_code": "330903",
      "release_time": 1635264000000
    },
    {
      "city": "舟山市",
      "level_title": "优秀",
      "district": "普陀区",
      "level_code": "A",
      "social_credit_code": "9133090307868393X1",
      "score_code": "8ca03c8e-634c-49cb-b35c-bac95ce46910",
      "ent_name": "浙江荣生海洋生物制品有限公司",
      "region_code": "330903",
      "release_time": 1635264000000
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list",
  "spider_start_time": "2021-10-27 11:28:51.445",
  "spider_end_time": "2021-10-27 11:28:58",
  "task_params":
  {
    "province": "zhejiang",
    "step": "start",
    "index": 2
  },
  "metadata":
  {
    "province": "zhejiang",
    "index": 2
  },
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.1.18"
}

福建：

{
  "data":
  [
    {
      "id": 11795,
      "social_credit_code": "91350504MA2Y13F352",
      "credit_year_batch": "2021年第二批",
      "ent_name": "泉州佰份佰卫生用品有限公司",
      "county": "洛江区",
      "city": "泉州市",
      "deptName": "泉州市洛江生态环境局",
      "createTime": "2021-06-03",
      "credit": "79",
      "credit_type": "环保良好企业"
    },
    {
      "id": 11794,
      "social_credit_code": "91350504MA346T4N1T",
      "credit_year_batch": "2021年第二批",
      "ent_name": "泉州洛江凤栖石材厂",
      "county": "洛江区",
      "city": "泉州市",
      "deptName": "泉州市洛江生态环境局",
      "createTime": "2021-06-03",
      "credit": "79",
      "credit_type": "环保良好企业"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list_of_normal",
  "spider_start_time": "2021-10-27 14:50:13.824",
  "spider_end_time": "2021-10-27 14:50:14",
  "task_params":
  {
    "province": "fujian",
    "step": "start",
    "index": 2020
  },
  "metadata":
  {
    "province": "fujian",
    "index": 2020
  },
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.6.51"
}

四川：

{
  "data":
  [
    {
      "id": 994222,
      "enterprise":
      {
        "id": 354497052,
        "name": "阆中市枣碧大梁山页岩机砖厂",
        "creditCode": "92511381MA695TYH83",
        "orgCode": "MA695TYH8",
        "enterpriseType": "PRIVATE_OWNED",
        "industry": "OTHER",
        "polluteType": "EXHAUST_GAS",
        "controlType": "CITY_PREFECTURE",
        "enterpriseAttr": "PRODUCTION_ENTERPRISE",
        "legalName": "庄兆雄",
        "legalTel": "15881493231",
        "productScale": "7.5万吨/年",
        "businessScope": "页岩砖生产、销售",
        "headOfEPA": "庄剑平",
        "regTime": "2008-05-10T00:00:00+08:00",
        "enterpriseState": "PRODUCTING",
        "regionCity":
        {
          "code": "511300000",
          "name": "南充市",
          "parent":
          {
            "code": "510000000",
            "name": "四川省"
          }
        },
        "regionDistrict":
        {
          "code": "511381000",
          "name": "阆中市",
          "parent":
          {
            "code": "511300000",
            "name": "南充市"
          }
        },
        "longitude": 105.2579,
        "latitude": 31.0116,
        "isSSGS": false,
        "enable": true,
        "lastModifiedDate": "2021-03-31T09:34:40.045+08:00",
        "address": "阆中市枣碧乡大梁山村"
      },
      "evaluateScore": 84,
      "selfScore": 98,
      "countyScore": 88,
      "cityScore": 84,
      "evaluateResult": "HBLHQY",
      "evaluateYear": 2020,
      "last": true,
      "publishOrg": "南充市生态环境局",
      "evaluateState": "GSZ",
      "veto": false,
      "dataFrom": "NormalCreditEvaluation"
    },
    {
      "id": 994221,
      "enterprise":
      {
        "id": 354496929,
        "name": "阆中市金福旺页岩机砖厂",
        "creditCode": "92511381MA62HXHX0D",
        "orgCode": "MA62HXHX-0",
        "industry": "OTHER",
        "controlType": "OTHER",
        "enterpriseAttr": "PRODUCTION_ENTERPRISE",
        "legalName": "金跃伟",
        "legalTel": "",
        "headOfEPA": "金光禄",
        "regionCity":
        {
          "code": "511300000",
          "name": "南充市",
          "parent":
          {
            "code": "510000000",
            "name": "四川省"
          }
        },
        "regionDistrict":
        {
          "code": "511381000",
          "name": "阆中市",
          "parent":
          {
            "code": "511300000",
            "name": "南充市"
          }
        },
        "isSSGS": false,
        "enable": true,
        "lastModifiedDate": "2020-07-10T12:29:10.841+08:00",
        "address": "阆中市江南镇瓦房沟村十四社"
      },
      "evaluateScore": 95,
      "selfScore": 104,
      "countyScore": 97,
      "cityScore": 95,
      "evaluateResult": "HBLHQY",
      "evaluateYear": 2020,
      "last": true,
      "publishOrg": "南充市生态环境局",
      "evaluateState": "GSZ",
      "veto": false,
      "dataFrom": "NormalCreditEvaluation"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list_of_normal",
  "spider_start_time": "2021-10-27 14:54:17.216",
  "spider_end_time": "2021-10-27 14:54:17",
  "task_params":
  {
    "province": "sichuan",
    "step": "start",
    "index": 1
  },
  "metadata":
  {
    "province": "sichuan",
    "index": 1
  },
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.1.18"
}

四川字段解析

湖南：

{
  "data":
  [
    {
      "SSQX": "雨花区",  # 区县
      "GSDJ": "环保合格单位", # 信用等级(与年度字段的年份相关)
      "ND": "2020", # 年度
      "QYMC": "长沙博大环保科技有限公司",  # 企事业单位名称
      "SSDS": "长沙市",  # 市州
      "GXSJ": "2021年09月27日",  # 更新时间
      "TYSHXYDM": "91430111344823182Y",  # 统一社会信用代码
      "CPDJ": 2,  # 参评等级
      "ZXDJ": "环保合格单位"   # 当前信用等级
    },
    {
      "SSQX": "保靖县",
      "GSDJ": "环保合格单位",
      "ND": "2020",
      "QYMC": "保靖县人民医院",
      "SSDS": "湘西土家族苗族自治州",
      "GXSJ": "2021年09月27日",
      "TYSHXYDM": "12433125448636058Q",
      "CPDJ": 2,
      "ZXDJ": "环保合格单位"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list",
  "spider_start_time": "2021-10-29 16:03:08.162",
  "spider_end_time": "2021-10-29 16:03:10",
  "task_params": {"province": "hunan","step": "start","index": 1},
  "metadata": {"province": "hunan","index": 1},
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.1.10"
}

河南：

{
  "data":
  [
    {
      "belongsBasin": "1",
      "companyAddress": "郑州市中原区桐柏南路158号",
      "companyLevel": "2",
      "companyName": "河南（郑州）中汇心血管病医院",      # 企业事业单位名称
      "contactAddress": "郑州市中原区桐柏南路158号",
      "contactNumber": "",
      "contactPerson": "",
      "contactTelphone": "15290405687",
      "contactUser": "周毅鹏",
      "contactWechat": "zyp15290405687",
      "controlLimit": "",
      "createTime": "2021-06-09 14:16:43",
      "createUser": "d857c4afdcc047f1a1fa70417df5f5f7",
      "emissionLimits": "",
      "emissionsTo": "",
      "enabledStatus": "1",
      "evaluateDate": "2021-09-24",                   # 评级时间
      "evaluation": "1",
      "exhaustType": "01,02,04",
      "finalResult": "警示",                          # 等级
      "hasInit": false,
      "id": "0351cdb38249462f8589d065f8a207af",
      "industry": "",
      "industryInvolved": "Q8415",
      "isUsed": "1",
      "legalRepresentative": "毛慧娟",                # 法人 
      "officePhone": "",
      "orgCode": "ceaa1f4652ae4d73a2bf82d36eb2e325",
      "orgName": "中原区生态环境局",                    # 评级单位
      "organizationCode": "52410100MJF72040XB",
      "outletName": "",
      "pKName": "id",
      "parentCode": "410100",
      "pollutantName": "",
      "postcode": "450000",
      "pregion": "郑州市",                            # 城市
      "productionDate": "2009-11-05",
      "region": "中原区",                             # 区县
      "regionCode": "410102",
      "regionName": "",
      "registeredAddress": "",
      "remarks": "",                                 # 备注
      "scores": "75.0",
      "unicode": "52410100MJF72040XB",               # 统一社会信用代码
      "updateTime": "2021-07-08 15:57:38",
      "updateUser": "cd83c18f52a847bdb3466dacebc80515"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list",
  "spider_start_time": "2021-10-28 15:31:48.890",
  "spider_end_time": "2021-10-28 15:33:17",
  "task_params":
  {
    "province": "henan",
    "step": "start",
    "city": "郑州市",
    "index": 1
  },
  "metadata":
  {
    "province": "henan",
    "city": "郑州市",
    "index": 1
  },
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.1.18"
}

湖北:

{
  "data":
  [
    {
      "序号": "1",
      "行政区域": "天门市",
      "单位名称": "天门市岳口新兴猪场",
      "单位地址": "天门市岳口镇新兴垸村三组",
      "法定代表人": "罗远松",
      "当前得牌": "蓝标",
      "时间": "2020-01-19 13:35:33.0"
    },
    {
      "序号": "2",
      "行政区域": "天门市",
      "单位名称": "天门市新跃养殖场",
      "单位地址": "天门市岳口镇新兴垸村",
      "法定代表人": "罗良洲",
      "当前得牌": "蓝标",
      "时间": "2020-01-19 13:35:33.0"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list",
  "spider_start_time": "2021-10-29 10:45:05.281",
  "spider_end_time": "2021-10-29 10:45:09",
  "task_params":
  {
    "province": "hubei",
    "step": "start",
    "index": 1
  },
  "metadata":
  {
    "province": "hubei",
    "index": 1
  },
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.1.10"
}

广东:

{
  "data":
  [
    {
      "序号": "1",
      "地区": "佛山市",
      "组织机构代码/社会统一信用代码": "914406056844813696",
      "企业名称": "佛山市南海区里水污水处理有限公司(和桂污水处理厂)",
      "年度评定结果_2019": "未参评",
      "年度评定结果_2018": "蓝牌",
      "年度评定结果_2017": "蓝牌"
    },
    {
      "序号": "2",
      "地区": "佛山市",
      "组织机构代码/社会统一信用代码": "",
      "企业名称": "佛山市海天（高明）调味食品有限公司",
      "年度评定结果_2019": "绿牌",
      "年度评定结果_2018": "绿牌",
      "年度评定结果_2017": "绿牌"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list",
  "spider_start_time": "2021-10-29 10:02:47.770",
  "spider_end_time": "2021-10-29 10:02:49",
  "task_params":
  {
    "province": "guangdong",
    "step": "start",
    "index": 2
  },
  "metadata":
  {
    "province": "guangdong",
    "index": 2
  },
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.1.10"
}

贵州:

{
  "data":
  [
    {
      "企业名称": "遵义市红花岗区口腔医院",
      "污染源地址": "遵义市红花岗区子尹路157号",
      "参评年度": "2015",
      "系统评分": "87",
      "系统评级结果": "B++"
    },
    {
      "企业名称": "贵州昊龙胜境建材有限责任公司",
      "污染源地址": "",
      "参评年度": "2015",
      "系统评分": "87",
      "系统评级结果": "B++"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list",
  "spider_start_time": "2021-10-29 14:08:28.227",
  "spider_end_time": "2021-10-29 14:08:29",
  "task_params":
  {
    "province": "guizhou",
    "step": "start",
    "year": 2015,
    "index": 3
  },
  "metadata":
  {
    "province": "guizhou",
    "year": 2015,
    "index": 3
  },
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.1.10"
}

广西:

{
  "data":
  [
    {
      "统一社会信用代码": "91451281340369170J",
      "企业名称": "广西万屹工贸有限公司",
      "行业分类": "化工",
      "行政区划": "河池市 宜州市",
      "法人代表": "韦泽达",
      "是否参评": "已参评",
      "最近评价时间": "2021-07-12",
      "信用等级": "普通",
      "是否有效": "有效"
    },
    {
      "统一社会信用代码": "9145128168 51842278",
      "企业名称": "广西鑫华源水务有限公司",
      "行业分类": "污水处理",
      "行政区划": "河池市 宜州市",
      "法人代表": "霍军",
      "是否参评": "已参评",
      "最近评价时间": "2021-07-12",
      "信用等级": "守信",
      "是否有效": "有效"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "list",
  "spider_start_time": "2021-10-29 15:13:45.615",
  "spider_end_time": "2021-10-29 15:13:48",
  "task_params":
  {
    "province": "guangxi",
    "step": "start",
    "index": 4
  },
  "metadata":
  {
    "province": "guangxi",
    "index": 4
  },
  "spider_name": "environmental_protection",
  "spider_ip": "10.8.1.10"
}

爬虫运行环境

scrapy

爬虫部署信息

target: node_51,
spider_name: environmental_protection
5个进程

Taskhub地址

提交任务地址: 
代码编写地址:

Taskhub调度规则说明

爬虫监控指标设计

(先观察，待补充)
索引: 
监控频率: 
监控起止时间: 
报警条件: 
报警群:  
报警内容:

数据归集

责任人

数据归集方式

爬虫直接写kafka
爬虫写文件logstash采集

爬虫结果目录

采集文件存放路径：
/data/gravel_spiders/environmental_protection

归集后存放目录

10.8.6.228:
/data2_227/grvael_spider_result/environmental_protection

logstash配置文件名称

logstash文件采集type

数据归集的topic

general-taxpayer

ES日志索引及筛选条件

gravel-spider-data-*

environmental_protection_grade

基本信息

数据名称（中文）

数据英文名称

采集网站（采集入口）

采集频率及采集策略

存量更新策略

增量采集策略

爬虫

责任人

爬虫名称

代码地址

队列名称及队列地址

优先级队列说明

任务来源

任务输入参数（样例）

任务样例

任务参数说明

data_type说明

爬虫结果的超级数据

实际爬虫结果的数据结构

江苏：

浙江：

福建：

四川：

湖南：

河南：

湖北:

广东:

贵州:

广西:

爬虫运行环境

爬虫部署信息

Taskhub地址

Taskhub调度规则说明

爬虫监控指标设计

数据归集

责任人

数据归集方式

爬虫结果目录

归集后存放目录

logstash配置文件名称

logstash文件采集type

数据归集的topic

ES日志索引及筛选条件

监控指标看板

数据保留策略

数据清洗

责任人

代码地址

部署地址

部署方法及说明

数据接收来源

数据存储表地址