Pass rate question to NCARB

Comments

6 comments

  • Avatar
    Nicholas Civitano

    Hi just curious if NCARB has given thought to this at all, thanks!

  • Avatar
    Yike Qin

    Hi Nicholas,

    I think your suggestion is valid, but I also want to propose another possible reason for the passing rate drop.

    There are several popular order people taking the exams

    1) As NCARB made the exam

    2) PPD and PDD first.

    3) PPD and PDD last.

    I did the exams in order 1, but I can imagine how daunting it would be if I took PPD or PDD as my very first exam. As a result, if  a person started with PPD and PDD, there may be a higher possibility to fail. Therefore, the overall percentage of passing drops. If follow this logic, it may also explains why CE's passing rate goes to that high. People rarely took this exam first. Usually, it would be after PcM and PjM, or after all other 5. With a lot hours in the testing center, and familiarity with what to expect, the passing rate will very likely goes high. 

    Since ARE is graded on a absolute scale, lower passing rate just means more people failing the exam. This does not tell too much information. Definitely like to see more detailed statistics, such as how many time peoples fails before passing a division, what are the order that people actually taking the exams, etc.

  • Avatar
    Nicholas Civitano

    Hey Yike Qin,

    Interesting thoughts on this. I did the exams as : PJM PCM CE PA PPD PDD in that order and only failed PPD. I was very confident going in and knew the way these tests are set up. I can definitely see numbers being skewed if candidates are taking PPD and PDD first as they are definitely the most difficult. However, I wonder as a general psychology of test taking how many people go in taking the longest tests first? Surely there are people who do it, I know on these forums others talking about doing it this way but it still seems like most people, especially those purely in 5.0 would do these big two last. I.e. they know the routine of studying the suggested material or whatever they have done to be successful on the first 4 before going into these bigger two.

    On that same thought, many people probably only have PPD and PDD left if they are transitioning and it seems like the format differences could also cause more fails on these two than the others.

    There are many, many reasons why it could be. I don't think this is a vast conspiracy (some others do, check the forums and other pages which talk about ARE testing) that NCARB wants people to fail to make more money. However I think it would be in NCARBS best interest to try to elaborate on why the pass rate is going down on these two tests. If not for any other reason to alleviate some stress for new candidates and others in the midst's of taking the exams. I'm currently restudying and trying my best to apply my testing experiences and real life experiences to give myself the best shot at passing it on my next retake tentatively scheduled for April.  

    I guess I still don't really understand why we don't get more feedback on these exams. The vague 1-4 score report doesn't really do anything. A lot of people I see failing the AREs more than once have shown that they will improve in one area only to get worse in an area where they had a 1 or 2 before. To me, this speaks more to the questions not really testing the knowledge rather just getting the right type of question or the right wording enough times. I think candidates are realistic in that they are not expecting a graded exam with right and wrong answers however knowing how close you are to passing or failing would also help guide our studying. In other words if I only failed PPD by a few questions I would know, ok I am doing pretty good on these sections or these types of questions  but these other ones I missed X questions so I better brush up more. Getting a score that says "3" in Building codes or whatever doesn't help to guide our research and studying at all. What if I get all the MC correct and get a 0/20 on the case study? What if I get the point and clicks right or the drag drops but do terrible on the MC? I guess being graded by the content areas, especially for PPD and PDD is just not enough in my opinion. Really, I think my whole point is that regardless of why the pass rates are going down, they are going down and that isn't good for anyone. It costs the candidates a lot in time and money and it makes NCARB look bad in terms of their test design strategy. Logical reasoning would suggest that NCARB should probably address this, even if its just a simple: We think you guys need to study more and we are ok with these pass rates.

  • Avatar
    Nick NCARB (Edited )

    Hey Nicholas,

    Thank you for the thoughtful post on ARE 5.0 pass rates! First, it is important to understand that when ARE pass rates are published by NCARB, they represent only a snapshot of how candidates performed within a limited testing window (in this case, 1 year). Also, these percentages include ALL candidate test administrations, which means candidates failing a division multiple times can have a significant impact on the overall divisional pass rate. At this point in time, we have not looked at first time tester vs retester pass rates but plan to do so in the coming months. I think it is safe to hypothesize there will be a difference between the two percentages.

    Regarding exam scoring, in early 2017, NCARB established cut scores for all six divisions of the ARE. I recommend reading this blog post on the cut score process if you haven't already. A cut score is a minimum standard against which everyone is measured independently, which means the ARE is not graded on a curve. If a candidate performs at or above the cut score in a division, they will pass that division, regardless of how other candidates perform (more on ARE scoring here). If more candidates in a given year meet the cut score, the pass rate is higher.  

    Now if we compare the PP&D and PD&D divisions in 2017 to 2018, it is critical to note that they are statistically identical in difficulty. Since both divisional pass rates were higher in 2017, this may suggest that candidates who tested in 2017 were more prepared for these divisions than candidates in 2018. What defines a candidate as prepared? It could be their education, experience, dedicated study time, or a combination of all three. Again, candidate performance within a testing window determines the pass rate. NCARB does not have a defined pass rate that we must meet for each ARE division. 

    I hope this helps clarify the latest pass rates a bit. You guys have made some great points in this thread, and we plan to break down these pass rates further in the coming months. 

    On the topic of score reports, check out this thread on the Community. In short, only limited information is provided on failing score reports. The intent of the ARE is not to teach you to pass the exam, rather, assess your knowledge and skill for professional licensure.  

  • Avatar
    Nicholas Civitano

    Hey Nick,

     

    Thanks for the reply, really do appreciate it.

    I understand completely on how the tests are graded. I also understand that NCARB isn't an educational institution and the intent isn't to provide information to help candidates study or learn from the exams. However, my point was to highlight that PPD and PDD have over the time they have been active shown their scores drop and the other exams haven't. To me, it is hard to imagine those transitioning or starting into ARE 5.0 at the outset were "more prepared" than those who have had the exam feedback on the forums, the additional study guide information, Black spectacles getting approved as a study material etc etc. Who knows, and to your point, yes it is entirely possible that more people are taking the exam and they just are not prepared. Whatever the case may be, the scores have dropped over the 3 "test years" (I understand its not 3 whole years).

    My suggestion was that perhaps although the tests are "statistically" equivalent in difficulty that if the pass rates (46%, for PPD) is not acceptable in NCARBs eyes then perhaps they need to look at the format, the questions, or maybe lower the cut score. Obviously NCARB has a bunch of data and professionals who build exams working for them so maybe there is something else that can explain it but it still does seem like the information provided as study material and the actual exam do vary quite a bit, depending on which form of the exam you get. 

    As for the score report, I also understand it is not supposed to be a teaching tool, but if you take my two attempts at PPD (which is partially why I am confused by this exam, I passed the other 5 on the first try and studied harder for PPD than the others, making the "not prepared" statement hard to swallow)I actually have received a "2" in all of the divisions, just over the course of two exams. Now technically, based on that report I have scored an acceptable level to pass in each division, but based upon my mix of questions, cut score or whatever I haven't been able to pull that together for one exam. If you see what I'm getting at it is that, it is difficult to prepare for an exam after studying for it twice, getting the equivalent "passing" score in all the divisions but just not on the same attempt. I don't really know what I am missing or what else to prepare for. I am literally currently studying as I write this, I have my material out and taking notes, practice questions and trying to apply this information referenced in the Ncarb study guide the best I can. However if the feedback I got had more information regarding what type of question I do well on or how many points I am off by then maybe I can learn to apply this knowledge in a way that Ncarb deems necessary to pass.

     

    Just some thoughts, I am scheduled to take the exam in April so assuming Ncarb was to change anything it probably wouldn't go into my test.

     

    One question though: How often are the test questions changed. I.e. say I was getting questions that have proven to be statistically too difficult, is there a chance they have been cycled out? Are questions continually cycled in and out or is it more of a once a year type of thing?

     

    Thanks

  • Avatar
    Nick NCARB (Edited )

    Hey Nicholas,

    Each division of the ARE has multiple versions in the test center, so if you take a division 3 times in a year, you will see a different version of the test each time. Typically, we switch out the versions of each division yearly. One thing to remember, questions that are statistically too difficult don't make it past the pretesting status on the exam. They are retired before they become scored items.  

Please sign in to leave a comment.

Powered by Zendesk