Image generated from ideogram
Today (March 16) is the 2024 civil servant written examination in Guangxi. I have heard some friends discussing the positions they are applying for.
Guotou thought: Now domestic AILarge ModelThere are so many, is it possible to use them to help me quickly interpret the information in the job list? Wen Xin Yi Yan 3.5,iFlytek Spark,Bean curd,Zhipu QingyanGLM-3, Zhipu Qingyan GLM-4,Thousand Questions on Tongyi,KimiWhich one is stronger?
I thought about my needs, which are as follows:
- Upload the job table excel file.
- Verify that AI can read the table data correctly.
- Do you know how many jobs there are in Nanning, Guangxi?
- How many positions in Nanning are eligible for "computer-related majors"?
- How many jobs are there in each city in Guangxi?
- Based on the job data of each city, can it be presented in a tabular form?
- Based on the job data of each city, can it be presented in the form of a bar chart?
- Based on the job data of each city, what is the respective proportion?
- Based on the proportion of job data in each city, can it be presented in the form of a pie chart?
The figure below shows the results of this evaluation. The full score is 45 points. The specific scoring criteria and evaluation process are shown below.
The picture comes from the Guotou evaluation results
Scoring Criteria
Before testing, in order to more intuitively and conveniently compare the test results of each model, we briefly define the test scoring criteria as follows:
|
Scoring Criteria |
Fraction |
|
As expectedExactly matches |
5 |
|
As expectedMostly consistent |
4 |
|
As expectedA few parts match |
3 |
|
Unable to determine whether it matches |
2 |
|
As expectedTotally inconsistent, irrelevant answer |
1 |
|
No such featureorDoes not support generation |
0 |
(Note: This score is only for the perceptual feedback of this test. It is not an authoritative test, so please don't worry about it.)
Test items
- Upload the form file.
- Read the value of a cell in a row and column in a data table.
- Read the statistics of a dimension in the data table.
- Read the statistics of two dimensions in the data table.
- Based on the understanding of the data table, multiple statistical data are output.
- Based on the understanding of the data table, multiple statistical data are output and converted into a tabular form.
- Based on the understanding of the data table, multiple statistical data are output and converted into a bar chart.
- Based on the understanding of the data table, output the proportion of multiple statistical data.
- Based on the understanding of the data table, output the proportions of multiple statistical data and convert the proportion data into a pie chart.
Testing the Model
- Wenxin Yiyan-Wenxin Big Model 3.5
- iFlytek Spark
- Thousand Questions on Tongyi
- Bean curd
- Zhipu Qingyan-GLM-4
- Zhipu Qingyan-GLM-3
- Kimi
Testing time
March 15 and 16, 2024.
After the preparation work is completed, we start to test.
Wenxin Yiyan-Wenxin Big Model 3.5
Test 1: Upload a table file
Feedback and Ratings:Only supports word and pdf, not xlsx and other table formats. Unable to conduct subsequent tests, ended before it even started-_-||
Total score: 0.
iFlytek Spark
Test 1: Upload a table file
Feedback and Ratings:Does not support xlsx and other table formats. Unable to conduct subsequent tests, ended before it even started -_-||
Total score: 0.
Bean curd
Test 1: Upload a table file
Feedback and Ratings:File upload is not supported. Subsequent testing is unavailable. It ended before it even started -_-||
Total score: 0.
Zhipu Qingyan-GLM-3
Test 1: Upload a table file
Feedback and Ratings:Does not support xlsx and other table formats. Unable to conduct subsequent tests, ended before it even started -_-||
Total score: 0.
Zhipu Qingyan-GLM-4
Test 1: Upload a table file
Feedback and Ratings:Successfully uploaded the position table xlsx file, this item will be scored 5 points.
Test Item 2: Read the value of a cell in a row or column in a data table
Feedback and Rating: The picture below is a screenshot of the table. After removing the header column names, the third column of the first row of the text is indeed "People's Courts at All Levels". This item is rated 5 points.
Test Item 3: Read the statistics of a dimension in the data table
Feedback and scoring: The figure below is a screenshot of the table. The Nanning sheet contains 524 positions. If the answer is correct, score 5 points.
Test Item 4: Read the statistics of two dimensions in the data table
I found that when I directly searched for keywords including "computer majors", nothing was found.
Let's ask it another way: How many jobs are there in Nanning that are suitable for computer majors? I found that 21 jobs can be retrieved.
Feedback and Ratings:The picture below is a screenshot of the table. A total of 21 professional categories containing "computer" were filtered out from the Nanning sheet. The answer is correct, but since the first search result was 0, it is considered partially in line with expectations and is scored 3 points.
Test Item 5: Based on the understanding of the data table, output multiple statistical data.
Feedback and scoring: The figure below is a screenshot of job statistics in some cities. If the answer is correct, you will get 5 points.
Test Item 6: Based on the understanding of the data table, output multiple statistical data and convert the statistical data into a tabular form.
Feedback and Ratings:Although the table header has an English name, the answer is correct and 5 points are awarded.
Test Item 7: Based on the understanding of the data table, output multiple statistical data and convert the statistical data into a bar chart.
Feedback and Ratings:A bar chart can be generated, but the city names cannot be displayed. This is considered to be a small part that meets expectations, so I give it a score of 3.
Test Item 8: Based on the understanding of the data table, output the proportion of multiple statistical data.
Feedback and Ratings:If the percentage answer is correct, 5 points will be awarded.
Test Item 9: Based on the understanding of the data table, output the proportions of multiple statistical data and convert the proportion data into a pie chart.
Feedback and Ratings:The pie chart proportion data is correct, but the option name cannot be displayed, which is considered a small part of the compliance and is scored 3 points.
Total score: 5+5+5+3+5+5+3+5+3=39.
Thousand Questions on Tongyi
Test 1: Upload a table file
Feedback and Ratings:Successfully uploaded the position table xlsx file, this item will be scored 5 points.
Test Item 2: Read the value of a cell in a row or column in a data table
Feedback and Ratings:It is unable to give an answer, and after further clarifying the input, it is still unable to give an answer. It also says that it needs to be guessed. This is considered to be irrelevant and completely inconsistent with the question. This item will be scored 1 point.
Test Item 3: Read the statistics of a dimension in the data table
Feedback and Ratings:I only read part of the information from the table and made some data based on it. I took the information out of context and answered the question irrelevantly. I will give you 1 point.
Test Item 4: Read the statistics of two dimensions in the data table
Feedback and Ratings:The result is the same as the third test. If you selectively extract part of the information to answer the question, take it out of context, or answer irrelevant questions, you will get 1 point.
Test Item 5: Based on the understanding of the data table, output multiple statistical data.
Feedback and Ratings:Similar to the previous test results, if you selectively extract part of the information to answer the question, take it out of context, or answer irrelevant questions, you will get 1 point.
Test Item 6: Based on the understanding of the data table, output multiple statistical data and convert the statistical data into a tabular form.
Feedback and Ratings:Based on the results of test item 5, it was correctly converted into a table format and the table presentation function met expectations, so it was given a score of 5.
Test Item 7: Based on the understanding of the data table, output multiple statistical data and convert the statistical data into a bar chart.
Feedback and Ratings:It does not support drawing graphs or bar charts. Without this function, this item is scored 0.
Test Item 8: Based on the understanding of the data table, output the proportion of multiple statistical data.
Feedback and Ratings:The data percentage requirement was raised, but the total number of people could not be directly identified. They kept asking me to provide the total number of people. Well, then we can use the information given as the total number of people, or there is no total number of people in all positions. OK, then I will give the total number of positions as 4210. At this time, they told me a lot of reasons, but I don't have the patience to waste time with them. This is irrelevant, so I give 1 point.
Test Item 9: Based on the understanding of the data table, output the proportions of multiple statistical data and convert the proportion data into a pie chart.
Feedback and Ratings:Does not support drawing charts, does not have this function, and gives 0 points.
Total score: 5+1+1+1+1+5+0+1+0=15.
Kimi
Test 1: Upload a table file
Feedback and Ratings:The first upload prompted that it exceeded the conversation length and was not supported. After deleting most of the content in the Excel table and retaining only the job plan table of Wuzhou City, it was successfully uploaded. This is considered a small part of the compliance, and this item is scored 3 points.
Test Item 2: Read the value of a cell in a row or column in a data table
Feedback and Ratings:The picture below is a screenshot of the table. After removing the header column name, the third column of the first row of the text is indeed "city level", which is 5 points.
Test Item 3: Read the statistics of a dimension in the data table
Feedback and Ratings:The picture below is a screenshot of the table. From the Wuzhou sheet, we can see that there are a total of 308 rows. After removing the header, there are a total of 307 job details. The answer is correct and is worth 5 points.
Test Item 4: Read the statistics of two dimensions in the data table
Feedback and Ratings:The first filter was "computer majors", and Kimi converted it into the keyword "computer science and technology", but the output was wrong. The second time, Kimi did not use quotation marks to convert it into mentions of "computer majors", but the statistical data and detailed descriptions were wrong. See the screenshot below, you can see that for position numbers 14-19, the word "computer" does not appear in the corresponding professional (discipline) category. This is considered a non-answer, and I give it 1 point.
Test Item 5: Based on the understanding of the data table, output multiple statistical data.
Feedback and Ratings:The figure below is the result of pivoting the Excel table. It is found that some data are accurate, some data are wrong, and a small part is in line with expectations, so it is scored 3 points.
Test Item 6: Based on the understanding of the data table, output multiple statistical data and convert the statistical data into a tabular form.
Feedback and Ratings:5 points if the data can be correctly converted into a table based on the data results and meets expectations.
Test Item 7: Based on the understanding of the data table, output multiple statistical data and convert the statistical data into a bar chart.
Feedback and Ratings:Unable to create chart, no such function, 0 points.
Test Item 8: Based on the understanding of the data table, output the proportion of multiple statistical data.
Feedback and scoring: If the answer based on the statistical percentage of existing results is correct according to the estimated results and meets expectations, it will be scored 5 points.
Test Item 9: Based on the understanding of the data table, output the proportions of multiple statistical data and convert the proportion data into a pie chart.
Feedback and Rating: Unable to create a chart, no such function, 0 points.
Total score: 5+3+5+1+3+5+0+5+0=27 points.
Evaluation Summary
The evaluation results are summarized in the following table:
From the evaluation results table we can see that:
Zhipu Qingyan GLM-4Table data processingThe experience effect is far ahead.
Next was Kimi, who was able to answer most of the data questions but was unable to generate charts.
Next, I come to Tongyi Qianwen. I can only say that it has the function of uploading spreadsheet files, but its IQ is not enough to correctly process and analyze the spreadsheet data.
Finally, there are Wenxin Yiyan Large Model 3.5, iFlytek Spark, Doubao, and Zhipu Qingyan GLM-3, which are now simply not supported.
Based on the current performance of Zhipu Qingyan GLM-4, would you use it to process spreadsheet files?
Looking forward to the re-evaluation after the major models of each manufacturer are updated (*^▽^*).