An Analysis of HOTS Level Questions on the Indonesian Objective Tests

This study aimed to describe HOTS-level questions in the objective tests at the end-of-semester Indonesian exam for class XII at SMA Negeri 3 Bengkulu Tengah. The researcher's description was based on Anderson & Krathwohl's theory. This research was a qualitative descriptive study. The data source in this study was in the form of documents for the end-of-semester exams for class XII Indonesian at SMA Negeri 3 Bengkulu Tengah, totaling 105 questions using documentation data collection techniques. The analysis showed that HOTS questions still tend to be few and have not fully used HOTS-level questions. 11 questions meet the characteristics of HOTS questions or the equivalent of 10% of the data used. 10% of HOTS questions were dominated by the C4 thinking process dimension (analyzing), which consists of 6 questions (6%), the C5 thinking process dimension (evaluating) 4%, and the C6 thinking process dimension (creating) 1%. For the knowledge dimension, the HOTS questions were dominated by questions that measure the conceptual knowledge dimension.


INTRODUCTION
The 2013 curriculum was designed with various improvements.One of the focuses of its improvement focused on assessment standards, namely, gradually adapting international standard assessment models (Hartiti, 2020).Based on the results of an international study by the Programme for International Student Assessment (PISA)) it shows that the achievements of reading, mathematical, and science literacy achieved by Indonesian students could be higher.In general, Indonesian students' ability is meager in (1) integrating information, (2) generalizing case by case into a standard solution, (3) formulating real-world problems into subject concepts, and (4) conducting investigations.With this problem, there is a need for system changes in learning and assessment (Partinem, 2019).Wirandani et. al. (2019) explained one of the mandatory abilities a teacher must have: to create and develop an evaluation tool for student learning outcomes.The evaluation is an activity to see whether the planned program has been achieved.Learning evaluation is one of the activities carried out by an educator to assess the extent to which students can accept E-ISSN: 2461-131x P-ISSN: 2406-8586 learning activities.Nurgiyantoro (2010) defines assessment as a process to measure the level of achievement of learning objectives that have been pursued in learning activities that have been carried out.The tool used to evaluate is the assessment instrument.Instruments are tools used to assess how far students can understand the subject matter the teacher or educator delivers.In the learning process, the assessment instrument that educators often use is the test instrument (Oktarina, 2020).
Tests as an assessment tool are used by educators to measure the achievement of learners' learning outcomes in the realm of knowledge (cognitive).Kadir (2015) mentioned that tests are a method that can be used in the context of measurement and assessment in education.The test is expected to provide an overview of how students solve a problem related to their daily lives, so it is necessary to develop a contextual stimulus (according to the actual circumstances in the student's living environment.One form of test that can be used as an assessment tool is an objective test.An objective test is a test whose scoring can be carried out with high objectivity.The score produced at the end of scoring on the work of an objective test taker is not fundamentally different and would be the same if the scoring was done by two or more proofreaders or by the same corrector who scored twice or more at different times (Djiwandono, 2008); objective tests are questions or tests in which the information or answers needed to answer the questions are readily available (Yuniar et al., 2015) & Nurgiyantoro (2010), defines an objective test as a short answer test that requires students to give short answers, even just by choosing specific codes that represent the alternatives to the answers provided.
At this time, educational institutions must prepare students' competence to welcome the 21st century.In the 21st century, the term 4C is known as 4C, a vital competency needed at that time.The term 4C consists of (1) critical thinking, (2) creativity, (3) collaboration, and (4) communication.One of the efforts to prepare students with the fundamentals of 4C as an essential competency in the 21st century is to conduct an assessment based on the HOTS concept.A teacher is expected to be able to lead students to use high-level thinking skills so that later, students are used to solving every problem they encounter in their environment.Lewis & Smith (1993) ( in Setiawati, 2019) revealed that a higher level of thinking occurs when a person acquires new information stored in memory and is interrelated or reimagines or expands that information to achieve a goal or finds possible answers in confusing conditions.The HOTS thought process dimension, based on Anderson and Krathwohl's (2001) taxonomy in (Widana, 2017), generally measures the ability in the realm of analyzing (analyzing-C4), evaluating (evaluating-C5), and creating/creating (C6).In addition to trying to prepare students for the 21st century, HOTS questions also play a role in improving the quality of assessment so that the quality of assessment increases.By improving the quality of assessment, the quality of education will also increase.
Applying Higher Order Thinking Skills (HOTS) requires excellent attention for educators to provide learning oriented to the HOTS concept.Nugroho (2018) states that teachers must understand that good learning is student-centered.Student-centered means also centered on problems and activities.In order to stimulate students to think at a higher level, students must take part in every problem-solving process in the learning process.Suppose the educator can apply the HOTS learning concept in learning activities.In that case, the next task of the educator is to hold an assessment tool in the form of a test by considering in advance whether the test given meets the eligibility standards that lead to high-level skills or Higher Order Thinking Skills (HOTS) because HOTS-oriented assessments are not something new for educators in carrying out assessments.
Analysis of HOTS questions has previously been carried out by Rochman and Hartoyo ( 2018)) (with the title High Order Thinking Skills (HOTS) Analysis of Taxonomy Analyzing Physics Problems.This research was conducted at a high school in Bengkulu Tengah Regency to give an idea of the HOTS evaluation tool in Bengkulu Tengah High School, even though the subjects are different.In addition, the difference between the author's research and the previous study is that the author not only focused the research on the analysis of question items on the dimension of the analytical thinking process (C4) only and the author also did not focus on the ability of children to do HOTS questions, but the author focused on the HOTS question items in the final exam of semester Indonesian students of class XII SMA N 3 Bengkulu Tengah by analyzing the entire question item based on the realm of cognitive and knowledge dimensions in Anderson & Krtahwohl's taxonomy, then describe the data on the HOTS level that has been analyzed.Another relevant research that the author refers to is research on HOTS questions that has also been carried out by Wirandani et. al. (2019), with the title Item Analysis of HOTS (High Order Thinking Skills) Questions on Class XII School Examination Questions for Indonesian Subjects at SMK An-Nahl.
With this relevant research, it is one of the backgrounds for the author to conduct this study.Another background to why the author designed the research item about HOTS is because many things need to be clarified about the concept of the HOTS question itself.Many think difficult questions are questions about the HOTS level; difficulty differs from higherorder thinking.
Fundamental to the concept of learning that currently leads to HOTS to prepare students to welcome the 21st century as the author has described, also supported by the results of relevant research that previous researchers have carried out, the author will carry out research on the HOTS level at SMA Negeri 3 Bengkulu Tengah.This school is one of the educational institutions in the Bengkulu Tengah district that has implemented the 2013 curriculum in its learning concept.In addition, SMA Negeri 3 Bengkulu Tengah is an educational institution in great demand by students in the Bengkulu Tengah area, especially for Pondok Kelapa, Pondok Kubang, and surrounding districts.This is because SMA Negeri 3 Bengkulu Tengah is a reference school.With the background of SMA Negeri 3 as a reference school, it is necessary to pay attention to the assessment tool that teachers will use as a student learning evaluation tool that not only attaches importance to cognitive values but also needs to integrate strengthening character education for students in communicating learning in schools to problems that exist in the student's living environment (Ariyana et al., 2018).

METHOD
The method used by the author in this study is qualitative descriptive analysis.The author of this study intends to understand and describe in more depth the phenomenon of education at SMA Negeri 3 Bengkulu Tengah related to the HOTS-based learning assessment tool.The author chose a descriptive qualitative approach because the research that the author will do will answer the formulation of the problem in more depth if the data is analyzed by applying a qualitative descriptive approach compared to using quantitative methods or other types of qualitative methods.As stated by Moleong (2017), qualitative research is research that intends to understand phenomena about what is experienced by the subject of the study, for example, behavior, perception, motivation, action, Etc., holistically and using description in the form of words and language, in a specific context.
In this study, the author will conduct a content analysis of the teacher-made test assessment tool in the final semester examination document for class XII students at SMA N 3 Bengkulu Tengah.The results of the analysis in this study focus on the analysis of HOTS When analyzing the final semester exam question documents, the author analyzes the entire question item without exception, then after obtaining data in the form of questions with the coverage of LOTS, MOTS, and HOTS following the characteristics of each question item in the document recording guideline instrument that has been developed, the author also describes the HOTS level question data that has been obtained to get answers to how the HOTS level questions developed by the teacher in the final exam test for semester Indonesian class XII students based on the characteristics of the HOTS questions found in the question items that have been analyzed.

Results
Based on the results of the analysis that the author has done on 105 question items, it shows that the HOTS questions are found in the final exam questions for the Indonesian semester of class XII SMA Negeri 3 Bengkulu Tengah with a relatively small number or have not fully used HOTS questions.The number of HOTS questions in the data was 11, which included the dimensions of the C4, C5, and C6 thought processes and measured the dimensions of conceptual and metacognitive knowledge.The HOTS question is dominated by questions that measure the dimensions of the C4 thought process, namely six questions, while for C5 and C6 questions, there are four questions and 1 question each.The following is presented a table of analysis results that have been grouped according to the level of the problem.Six questions are included in the HOTS-level questions with the dimension of the C4 (analyzing) thought process.The questions consist of code questions 01/1, 03/1, 49/1, 05/2, 14/2, and 06/3.Based on the analysis results, only the dimensions of the C4 thought process were present in the three question documents.
These C4 questions have the following form of questions; 1) analyze the main idea of a paragraph; 2) examine sentences that are not solid in paragraphs; 3) sum up the sentence of the reviewer of the section of excellence in the text of the story presented; 4) analyze the similarity of intrinsic elements in two different story texts.The characteristics of the HOTS question that are fulfilled in the question code, in addition to measuring the dimension of the C4 thought process, are; 1) all question codes measure the dimensions of conceptual knowledge; 2) measuring critical and creative thinking skills in making decisions; 3) measure the transferability of one concept to another; 4) measure the ability to examine ideas and information critically; 5) measure the ability to use information to solve problems; 6) none of the existing questions have their answers expressly presented in the stimulus.

HOTS Level Questions with C5 Thinking Process Dimensions
Based on the analysis results, four questions include HOTS questions with the dimension of the C5 thought process.The question consists of code questions 14/1, 50/1, 17/2, and 33/2.This means that the problem comes from data 1 and data 2.This type of question is a question that instructs students to make an assessment based on specific criteria or standards.This type of question already meets the criteria for other HOTS questions, such as; 1) measuring the dimensions of conceptual and metacognitive knowledge; 2) measuring the ability to think critically and make decisions; 3) measuring the ability to examine ideas and information critically; 4) none of the question answers are expressly displayed in the stimulus; 5) measure the transfer ability of one concept to another; and 6) measure the ability to process and integrate information.The following will be presented as a question and a discussion of the analysis results.
These C5 questions have the following form of questioning; 1) assess the appropriateness between what is in the text and the context of everyday life; 2) assess the sentence of criticism corresponding to the quote of the short story presented; 3) criticize the choice of words that the author uses in his poems.

HOTS Level Questions with C6 Thinking Process Dimensions
The HOTS question identified in the question that measures the dimensions of the C6 thought process contains only one question in Document 3, namely the UAS question document for the 2020/2021 school year.The question is a question with code 01/3.
Questions that belong to this type are questions that meet the characteristics; 1) measuring the dimensions of conceptual knowledge; 2) measuring the dimensions of the C6 thought process because it instructs learners to create, i.e., composing the opening sentence of a cover letter based on a predetermined topic; 3) measuring critical thinking skills in making decisions; 4) measure the ability to analyze ideas and information critically; and 5) measure the ability to find links from a variety of different information.

Discussion
HOTS-based questions are instruments used to measure higher-order thinking skills, namely skills that do not just recall, restate, or refer without reciting.HOTS questions in the context of assessment measure skills: 1) transfer one concept to another, 2) process and integrate information, 3) find links from a variety of different information, 4) use the information to solve problems, and 5) critically examine ideas and information (Directorate of High School Development, 2017).
The HOTS problem in Anderson & Krathwohl's theory can be assessed through the dimension of the thought process and its knowledge.For the thinking process dimension, the HOTS level question category must be able to stimulate students to arrive at the C4 (analyzing), C5 (evaluating), and C6 (creating/ creating) thinking levels.The knowledge dimension is a HOTS question if the question measures the conceptual, procedural, and metacognitive dimensions of knowledge.If, in its application, there is a question that measures the dimensions of the C4-C6 thought process.However, if the question only measures the dimension of factual knowledge, then the question is categorized as a LOTSlevel question or low-level thinking.This happens because the factual knowledge dimension is characteristic of the LOTS level question, so when paired with any level of thought process from the C1-C6 level of thinking, the question will be identified as a LOTS level question.
Based on the author's research, the results are obtained that the HOTS questions in the UAS question document Indonesian class XII SMA Negeri 3 Bengkulu Tengah still need to be used.The number of HOTS questions that exist is only 11 questions.Syaifuddin (2019), in the Higher Order Thinking Skills Module Indonesian, mentioned that the government had set the percentage of HOTS questions in the exam at 10%.By this theory, it can be judged that the percentage of HOTS question items in the analyzed data has met the minimum percentage set by the government, which is 10% (11 question items) of the 105 questions analyzed.
The analysis results for the dimension of knowledge measured by the HOTS level are dominated by questions that measure conceptual knowledge by 9% (10 questions) and for the metacognitive knowledge dimension by 1% (1 question item).Widana (2017) explained that questions with dimensions of the C4 to thought process C6, which measures knowledge's conceptual, procedural, and metacognitive dimensions, constitute a category of higher-order thinking skills.Although the HOTS question is expected to measure the metacognitive knowledge dimension, based on this theory, the HOTS question that has been analyzed follows the characteristics of the HOTS question based on the knowledge dimension because there is no question with the C4 to C4 thinking process dimension-C6, which measures the dimensions of factual knowledge.
Overall, the final semester exam questions analyzed consisted more of MOTS-level questions.This contradicts the results of relevant research from Wirandani et. al. (2019).The relevant research results show results dominated by HOTS-level questions and only a few for LOTS questions.However, the analysis results regarding the dimensions of the thought process are already relevant.Namely, they are equally dominated by HOTS questions with the level of the C4 (analyzing) thinking process dimension.
1. HOTS Level Questions with C4 Thinking Process Dimensions Based on the results of the analysis of HOTS questions with the C4 thought process dimension, it is a question that includes the following questions; 1) analyze the main idea of a paragraph; 2) examine sentences that are not solid in paragraphs; 3) summing up the sentence of the reviewer of the section of excellence in the text of the story presented; 4) analyze the similarity of intrinsic elements in two different story texts.This is in line with the theory put forward by Brookhart (in the Center for Educational Assessment, 2019) that the HOTS question that measures the dimensions of the C4 thought process is a question that measures the ability to parse information into parts and determine or explain how the parts are related.The problem with measuring the analysis is when the participant must conclude based on the analysis of parts of the text or stimulus.An example of a question is to find or determine the main ideas, arguments, and assumptions of a text that are not explicitly conveyed; determine or compile evidence that supports and does not support a case description; determine the views of the author of the essay from a certain point of view.Moreover, the principles of HOTS require students to use and manipulate information and ideas by transforming meaning and implication (Parlan & Rahayu, 2021).It reflects when the students combine the facts and their ideas for synthesis, generalization, explanation, and proposed hypothesis.
The HOTS C4 question that has been analyzed is a question that measures the dimensions of conceptual knowledge of all.These questions use the Operational Verb (KKO) to analyze and conclude.This corresponds to the verb realm of knowledge from Anderson & Krathwohl of Widana (2017), which groups KKO analyzing and inferring into KKO for the C4 thought process dimension.

HOTS Level Questions with C5 Thinking Process Dimensions
Brookhart explained that the C5 dimension of the thought process is a question that measures the ability to evaluate according to the objectives; make judgments based on standards or criteria (Center for Educational Assessment, 2019: 3-4).Based on this theory, the HOTS question that measures the dimensions of the C5 thought process in the data source is a question that asks students to give an assessment based on the text presented.So the questions analyzed are by the existing theory.Namely, students are asked to assess the discourse text presented fundamentally on specific provisions, for example, the relationship of the discourse context with daily life, or assess the relationship between the discourse context and specific theories.
The operational verbs used in the 14/1, 50/1, 17/2, and 33///2 code/questions already correspond to the operational verbs that have been grouped by Anderson & Krathwohl.The operational verb assessing and criticizing is the KKO for the HOTS C5 question.As for the dimension of knowledge, in the HOTS C5 question, there is one question that measures the dimension of metacognitive knowledge.Following Widana's theory (2017) in the Higher Order Thinking Skill ((HOTS)) Problem Preparation Module, which explains that the metacognitive knowledge dimension is a problem that stimulates students to combine several subject areas.Referring to the theory, one question measures the dimensions of metacognitive E-ISSN: 2461-131x P-ISSN: 2406-8586 39 knowledge because the question asks the learner to connect his understanding of the field of language with everyday life.It arises when the students face unusual, uncertain questions or dilemma problems (Parlan & Rahayu, 2021;Aziz & Rawian, 2022).The concept of HOTS produced explanations, decisions, performances, and results in the context of students' knowledge and experiences.

HOTS Level Questions with C6 Thinking Process Dimensions
The HOTS question with the C6 thought process dimension is a 01/3 code problem.The form for this type of question is "to arrange the opening sentence of the job application letter according to the context of the advertisement presented at the stimulus."Assessing the C6 question measures the skill of critically studying ideas and information.It measures the skill of finding links from a variety of different information.Moreover, It is the skill to think about objects, contents, or problems thought by the thinker in improving the quality of his thinking by skillfully taking the structural content inherent in thinking and imposing intellectual standards on them (Muhibbuddin et al., 2023).So, to solve these questions, students must use the ability to think critically in making decisions because the alternative answers are all following the context of the stimulus presented.Widana (2017) mentioned that the HOTS C6 question is a question that asks students to compile, produce and create something based on a predetermined context.Based on this theory, the 01/3 code question is categorized as a HOTS C6 question because the question asks students to produce and create the opening sentence of a job application letter based on the context that has been given.The question also uses a "compiling" KKO, which is a sign that the question is a matter of HOTS C6 level and is analyzed based on the use of KKO.In addition, It tends to be essential in developing lifelong learning that enables learners to respond effectively to 21st-century demands (Retnawati, 2018).It is also essential to consider that the commitment towards HOTS came in line with the development of information and technology, where learners are in need different competencies to cope with the massive amount of information, such as analysis, synthesis, and evaluation (Qasrawi & Beniabdelrahman, 2020).

CONCLUSION
Based on the results of the analysis and discussion that the author has described, some are included in the HOTS-level questions from the data sources analyzed.The presence of the HOTS level question is 11 questions or 10%.This number can still be minimal compared to the presence of MOTS-level questions, which amounted to 53 questions, and LOTS-level questions, which amounted to 41.
The existing HOTS question is dominated by the HOTS-level question, which measures the dimensions of the C4 thought process.Of the 10% of HOTS questions analyzed, 6% were HOTS questions that measured the dimensions of the C4 thought process.4% is a HOTS-level question that measures the C5 thought process dimension and another 1% is a HOTS-level question that measures the C6 thought process dimension.In addition to being dominated by the C4 dimension of the thought process, the HOTS question is also dominated by questions that measure the dimension of conceptual knowledge.
The results of the author's research provided knowledge for several people with disabilities, such as teachers and schools, as a consideration in developing HOTS-based assessment instruments.Suppose teachers have optimized the preparation of HOTS questions.In that case, the quality of education will also improve because, with the HOTS-based E-ISSN: 2461-131x P-ISSN: 2406-8586 40 evaluation tool, students will be accustomed to facing the development of 21st-century education.
E-ISSN:2461 -131x  P-ISSN: 2406-8586   -8586    35 questions, so the author will describe the characteristics of each HOTS question item found in the final semester examination documents of class XII students at SMA N 3 Bengkulu Tengah which includes three different teaching years, namelyFY 2017FY   /2018FY  , 2018FY   /2019FY  , and 2020FY    /2021.  .

Table 1 .
Results of Analysis of Problem Items Based on Anderson & Krathwohl Theory