On piece of advice I always give about studying for certification exams is to write your own questions for any exam you are studying for. It’s a great exercise to put yourself in the mindset of the SMEs writing the questions and will give you a new angle on studying.
But it begs the question, how do you go about writing an exam question?
/start tl;dr
- Take one objective from the list of Exam Topics.
- Look at the bullet points and tools listed in Exam Topics and determine one piece of information to test on.
- Write a question with either 4 possible answers (where either 1 or 2 are correct) or 5 possible answers (where 3 are correct).
- The question should be “understanding” or “applying” knowledge not just rote memorization.
- All answers must be plausible (not made up).
- No responses should include key words from the question unless they all do
- Look at VMware’s sample items for suggestions but realize those items were not good enough to be used on the live exam.
- Post your own questions in the comments for feedback
/end tl;dr
A good exam question (and thus a good exam) is reliable, fair and valid. A reliable question means those candidates expected to pass the exam usually get the question right, a fair question means there are no biases inherent in the question, and a valid question is one that fitsthe criteria for the exam and is useful to determine if the right people pass.
The phases of exam design are:
JTA = Job Task Analysis
IDW = Item Development Workshop
Beta = Initial evaluation of items
Live
Job Task Analysis (JTA)
This is where the exam is conceptualized. Why is the exam needed? What will be on the exam? What sort of person will pass the exam? What knowledge will they have? What knowledge won’t they have?
Those last three questions define the Minimally Qualified Candidate or MQC. The person who would score a “300” on the exam
If you looked at the Exam Blueprints VMware used to make available for certification exams there was an “Intended Audience” section that summarized the MQC.
For the VCP550 exam it stated:
A candidate for the VCP-DCV certification has approximately six months experience working with a vSphere
implementation. They are typically infrastructure personnel who are capable of installing and configuring ESXi
hosts and can use VMware vCenter™ Server to monitor, manage, troubleshoot and administer virtual
machines. The successful candidate will most likely have additional industry-recognized general IT
certifications or the equivalent experience (typically 2-5 years).
The SMEs writing the exam questions get access to the full description with a list of tasks the MQC is expected to perform without help, a list of tasks performed with help and and a list of tasks an MQC is not expected to perform.
This helps narrow down the scope of the questions and help target the relative difficulty level of the questions.
The JTA will also develop the blueprint, the framework of topics and objectives that define the knowledge domain of the exam. This internal blueprint will eventually distill into the “Exam Topics” on the certification pages.
Once the blueprint is written there will be surveys of the SMEs to determine weighting. Weighting determines the ratio of how many points (questions) each section receives relative to the others.
For example, if the exam was on Guitars and you had topics of Electric (Objectives: solid body, hollow body) and Acoustic (Objectives: six string, twelve string) the survey might determine that 40% of all questions be on solid, 10% be on hollow, 45% be on six and 5% be on twelve.
Then when the number of questions on the exam is set, you can multiply to determine how many questions from each section will be on the exam.
Item Development Workshop (IDW)
Once you have the blueprint and the MQC description you can pull together SMEs and start writing questions.
Questions on VMware’s multiple choice tests are allowed to be 1-of-4, 2-of-4 or 3-of-five. There are no “all of the above” or “none of the above” or “a and b” questions per psychometrician guidelines.
To write a question, you take one of the objectives, review it’s details and any part of the MQC statement that might apply to it (For that exam on Guitars, maybe candidates are expected to know basic wiring schematics but not how to solder or perform “advanced” modifications) and find a single piece of information to write a question on.
A good question tests one piece of knowledge. It may have multiple components but the underlying assumption is there is only one question being asked. For example you don’t want to ask “which of the following is true” and then list four completely different things.
A question’s difficulty has multiple components, including cognitive complexity, technical depth and distractor quality. Cognitive complexity gets a little deep but basically for a VCP-level exam you are looking for mostly “Understand and apply” level questions and for VCAP mostly “Analyze and Evaluate”. See the link above for more information.
Note that cognitive complexity helps explain why “Which of these ports is used by vCenter?” is bad (that is a “remember” level question) while a graphical troubleshooting question about firewall rules can be good even when both require you to know that port 902 is used by vCenter. The second question requires that you understand how the communication works and not just have a list of ports memorized.
Technical depth is described in the MQC statement and is then interpreted by the SMEs writing the questions. There is some flexibility here, especially depending on who is writing and reviewing the questions. However, the beta and healthcheck processes are very good at identifying questions that are too easy or too hard.
Distractor quality refers to how easy it is to pick out the right answer. The point is for the MQC to quickly choose the right answer and go on but someone who is below the MQC level to not be able to guess the correct answer or even to easily reject one or more options.
For example, Which production car is faster? a) Bugatti Veyron Super Sport, b) Kia Soul, c) Nisan Versa, d) Volkswagon Golf
Or Which production car is faster? a) Bugatti Veyron Super Sport, b) SSC Ultimate Aero, c) Koenigsegg CCR, d) McLaren F1
The first one should be be pretty easy for pretty much anyone – among other things the correct answer is the longest but the second one you either know the correct answer or you have to guess.
Possible options should be plausible (nothing made up), they should all be either the same approximate length or all different lengths, and they should not include words from the stem (unless all options contain the word).
There should be no “teaching” in the question. For example:
“All cars should be started in Neutral or Park. What is the first thing to check if your lights dim when trying to start a car?” The first sentence is a “teaching” statement that adds no exam value and could instead be used to answer this question or a different question.
After the items are written and tech-edited they are thrown in a hopper and presented to beta candidates. The beta exam contains more questions than a normal exam and all of the items are “beta” status – none of them count right away. The mix of questions should align to the blueprint ratios determined by the SME survey mentioned above.
After a number of exams have been completed, analysis can begin.
Standard Setting
This involves lots of math but the idea is to come up with a sub set of questions presented to the beta candidates that the psychometricians state are all reliable, fair and valid questions. That list of questions is then run through a Standard Setting meeting where a group of SMEs is shown the questions and determine (through a series of surveys and discussions) what score a Minimally Qualified Candidate would achieve on that exam. That result is then used to determine pass/fail on the beta exam and that score and bank of validated questions is used to set up the live exam.
Note that on the live exams not all questions count for the passing score, a subset of questions are “unscored.” This allows new items or changed items to be tested without needing a standalone beta.
As items are randomly selected from the unscored pool, this can explain why some students report an excessive number of question on an obscure or lesser-known topic. A topic that received 2% of the weighting and this two questions on a 110-question exam might actually have 10 questions if the luck of the draw pulled 8 from the unscored pool.
Post your own questions in the comments for feedback!
Questions examples taken/modified from the 2V0-621D practice set at vmware.com
Good question:
Q:
After a host failure, a virtual machine has not restarted. Which two reasons could be the cause? (Choose two).
A:
Virtual machine was not protected by HA at the time of the failure
Virtual machine is using an RDM
Virtual machine does not have VMware Tools installed
Insufficient spare capacity on available hosts
Now, I would have liked to see “virtual machine” added to the last option to ensure the clang is evenly spread, but it’s a good question.
So-so question:
Q:
A network administrator wants to take advantage of vSphere Network I/O Control 3 (NIOC3). What version must the vSphere Distributed Switch be at to support NIOC3?
A:
6.0
5.5
5.1
5.0
The problem is it’s really a “remember” level question with some window dressing
Bad question:
Q:
An administrator is noticing that SCSI reservations are causing an ESXi host to have very slow performance. Which two solutions could eliminate potential sources of SCSI reservation conflicts? (Choose two).
A:
Reduce the number of snapshots for SCSI VMDKs
Upgrade the host to the latest BIOS
Downgrade the ESXi host version
Use Storage vMotion to convert all SCSI disks to Thin Provisioned
I tweaked this one to add clang. Note the “SCSI” references in the options.
Bad question:
Q:
After configuring Storage I/O controls and setting congestion thresholds, an administrator notices that Storage I/O is not functioning correctly. What could be the cause of this?
A:
VMDKs are larger than 2TB
Storage I/O is not licensed
Virtual machines are not enabled for Storage I/O
Storage Performance APIs are not installed
I tweaked this one to add the bogus last option. You can’t have purely fictitious options.
