问题描述

向拍拍发送POST请求，获取的数据出现部分中文乱码和记录数很少

向拍拍发送POST请求，获取的数据出现部分中文乱码和记录数很少？在在线工具测试的时候，能够获取大量的数据，一般有上百万的命中数，返回一页大概40条记录，但是现在获取的数据命中数hitNum都是18，返回的数据也都是低于18。请各位大神多多指点！！！！！感激不尽。附上源代码和部分获取的数据。
源代码：private String postRequest(String url, HashMap cookies, String protocol) throws OpenApiException {
// url=url+"?charset="+charset;
// System.out.println(url);

    HttpClient httpClient = new ContentEncodingHttpClient();
    //HttpClient httpClient = new DefaultHttpClient();
    //HttpClientParam p = httpClient.getParams();
    //p.setContentCharset("UTF-8");
    httpClient.getParams().setParameter(CoreProtocolPNames.HTTP_CONTENT_CHARSET, "UTF-8");
    HttpPost post = new HttpPost(url);
    post.addHeader("User-Agent", "PaiPai API Invoker/Java " + System.getProperty("java.version"));
    post.getParams().setParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, Boolean.FALSE);
    //设置请求超时
    httpClient.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 8000);
    //设置读取超时
    httpClient.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT,8000);
    try {

        List<NameValuePair> parameters = new ArrayList<NameValuePair>();
        NameValuePair pair;
        Set<String> keySet = params.keySet();
        for (String key : keySet) {

/** if("charset".equalsIgnoreCase(key)){
// continue;
// }
// pair = new BasicNameValuePair(key, params.get(key));
// parameters.add(pair);*/
Object obj = params.get(key);
// if(obj instanceof Arrays){
if(obj.getClass().isArray()){
String arr[] = (String[]) obj;
for (String value : arr) {
pair = new BasicNameValuePair(key, value);
parameters.add(pair);
}
// pair = new BasicNameValuePair(key, getValue(obj));
// parameters.add(pair);
}else if(obj instanceof String){
pair = new BasicNameValuePair(key, (String)obj);
parameters.add(pair);
}else{
System.out.println(obj.toString());
throw new RuntimeException("http get not support parameter");
}
}
HttpEntity entity = new UrlEncodedFormEntity(parameters, HTTP.UTF_8);
//entity.setContentEncoding("UTF-8");
post.setEntity(entity);

        HttpResponse response = httpClient.execute(post);
        if(response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) {
            return "HTTP协议出错：" + response.getStatusLine() + "。";
        }

        HttpEntity responseEntity = response.getEntity();

        String lastResponseContentType;
        byte[] lastResponseContent;
        if (responseEntity != null) {
            Header header = responseEntity.getContentType();
            if(header!=null)
                lastResponseContentType = header.getValue();
            else
                lastResponseContentType = "text/html";
            InputStream stream = responseEntity.getContent();
            try {
                ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
                byte[] buffer = new byte[3072];

               /* int c=-1;
                while((c=stream.read())!=-1){
                    byteStream.write(c);
                }*/
                int len = 0;
                while ((len = stream.read(buffer)) != -1)
                {
                    byteStream.write(buffer, 0, len);
                }
                lastResponseContent = byteStream.toByteArray();
                System.out.println("response返回的原始数据:"+lastResponseContent);
                return (new String(lastResponseContent, charset));
            }catch (RuntimeException e) {
                post.abort();
                throw e;
            }finally {
                stream.close();
            }
        }else{
            lastResponseContentType = null;
            Header header = response.getFirstHeader("Content-Type");
            if(header != null) {
                lastResponseContentType = header.getValue();
            }
            lastResponseContent = null;
        }
        return (lastResponseContentType);

    } catch (ClientProtocolException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }

    return "";
}

部分获取的数据：
06-28 01:19:08.282: I/System.out(278): "title": " **2013新款女装短袖百褶 裙修身雪纺连衣裙??",**
06-28 01:19:08.282: I/System.out(278): "nickName": "??????? ",
06-28 01:19:08.282: I/System.out(278): "uin": 943602899,
06-28 01:19:08.282: I/System.out(278): "leg4Status": 0,
06-28 01:19:08.282: I/System.out(278): "leg3Status": 0,
06-28 01:19:08.282: I/System.out(278): "leg2Status": 0,
06-28 01:19:08.282: I/System.out(278): "leg1Status": 0,
06-28 01:19:08.282: I/System.out(278): "colStatus": 0,
06-28 01:19:08.282: I/System.out(278): "autoStatus": 0,
06-28 01:19:08.282: I/System.out(278): "cashStatus": 0,
06-28 01:19:08.282: I/System.out(278): "officStatus": 0,
06-28 01:19:08.282: I/System.out(278): "userVaStatus": 0,
06-28 01:19:08.282: I/System.out(278): "crValue": 0,
06-28 01:19:08.282: I/System.out(278): "cvValue": 0,
06-28 01:19:08.282: I/System.out(278): "level": 0,
06-28 01:19:08.282: I/System.out(278): "price": 10500,
06-28 01:19:08.282: I/System.out(278): "colorInfo": "",
06-28 01:19:08.282: I/System.out(278): "hongBaoInfo": "2013新款女装短袖百褶裙修身雪纺连衣裙??",
06-28 01:19:08.282: I/System.out(278): "property": 17179870356,
06-28 01:19:08.282: I/System.out(278): "saleNum": 0
06-28 01:19:08.282: I/System.out(278): }
06-28 01:19:08.282: I/System.out(278): ],
06-28 01:19:08.282: I/System.out(278): "pageNum": 2,
06-28 01:19:08.292: I/System.out(278): "pageSize": 20,
06-28 01:19:08.292: I/System.out(278): "resultSize": 17,
06-28 01:19:08.292: I/System.out(278): "hitNum": 18
06-28 01:19:08.292: I/System.out(278): }}

解决方案

抓包对比一下，看两个请求之间的数据区别